Diagnostic intraoral scanning

ABSTRACT

Methods and apparatuses for taking, using and displaying three-dimensional (3D) volumetric models of a patient&#39;s dental arch. A 3D volumetric model may include surface (e.g., color) information as well as information on internal structure, such as near-infrared (near-IR) transparency values for internal structures including enamel and dentin.

CROSS REFERENCE TO RELATED APPLICATIONS

This patent application claims priority to U.S. provisional patentapplication No. 62/622,798, titled “DIAGNOSTIC INTRAORAL SCANNERS,”filed on Jan. 26, 2018, and U.S. provisional patent application No.62/758,503, titled “DIAGNOSTIC INTRAORAL SCANNERS,” and filed Nov. 9,2018, each of which is herein incorporated by reference in its entirety.

This patent application may be related to one or more of: U.S. patentapplication Ser. No. 15/662,234, filed Jul. 27, 2017, titled “INTRAORALSCANNER WITH DENTAL DIAGNOSTICS CAPABILITIES”, which claimed priority toU.S. Provisional patent applications No. 62/367,607 (filed Jul. 27,2016) and 62/477,387 (filed Mar. 27, 2017); U.S. patent application Ser.No. 15/662,250, filed on Jul. 27, 2017, titled “METHODS AND APPARATUSESFOR FORMING A THREE-DIMENSIONAL VOLUMETRIC MODEL OF A SUBJECT'S TEETH”,which claimed priority to U.S. Provisional patent applications No.62/367,607 (filed Jul. 27, 2016) and 62/477,387 (filed Mar. 27, 2017);and U.S. patent application Ser. No. 15/672,248, Filed on Aug. 8, 2017,and titled “METHODS FOR DENTAL DIAGNOSTICS”, which claimed priority toU.S. Provisional patent applications No. 62/367,607 (filed Jul. 27,2016) and 62/477,387 (filed Mar. 27, 2017). Each of these application isherein incorporated by reference in its entirety.

INCORPORATION BY REFERENCE

All publications and patent applications mentioned in this specificationare herein incorporated by reference in their entirety to the sameextent as if each individual publication or patent application wasspecifically and individually indicated to be incorporated by reference.

BACKGROUND

Many dental and orthodontic procedures can benefit from accuratethree-dimensional (3D) descriptions of a patient's dentition andintraoral cavity. In particular, it would be helpful to provide athree-dimensional description of both the surface, and internalstructures of the teeth, including the enamel and dentin, as well ascaries and the general internal composition of the tooth volume.Although purely surface representations of the 3D surfaces of teeth haveproven extremely useful in the design and fabrication of dentalprostheses (e.g., crowns or bridges), and treatment plans, the abilityto image internal structures, including the development of caries andcracks in the enamel and underlying dentin, would be tremendouslyuseful, particularly in conjunction with a surface topographicalmapping.

Historically, ionizing radiation (e.g., X-rays) have been used to imageinto the teeth. For example, X-Ray bitewing radiograms are often used toprovide non-quantitative images into the teeth. However, in addition tothe risk of ionizing radiation, such images are typically limited intheir ability to show features and may involve a lengthy and expensiveprocedure to take. Some intraoral features such as soft tissues, plaqueand soft calculus may not be easily visualized via x-ray because oftheir low density. Other techniques, such as cone beam computedtomography (CBCT) may provide tomographic images, but still requireionizing radiation.

Thus, it would be beneficial to provide methods and apparatuses,including devices and systems, such as intraoral scanning systems, thatmay be used to model a subject's tooth or teeth and include bothexternal (surface) and internal (within the enamel and dentin)structures and composition using non-ionizing radiation. The model ofthe subject's teeth may be a 3D volumetric model or a panoramic image.In particular, it would be helpful to provide methods and apparatusesthat may use a single apparatus to provide this capability.

SUMMARY OF THE DISCLOSURE

Described herein are methods and apparatuses for taking, using anddisplaying dental information including information extracted fromthree-dimensional (3D) volumetric models of a patient's dental arch. A3D volumetric model may include surface (e.g., color) information aswell as information on internal structure, such as near-infrared(near-IR) transparency values for internal structures including enameland dentin. In some variations, the 3D volumetric scan may include or bederived from one or more other scanning modalities, including, but notlimited to: optical coherence tomography (OCT), ultrasound (US),magnetic resonance imaging (MRI), X-ray, etc.

In particular, described herein are methods and user interfaces fordisplaying and manipulating (e.g., sectioning, marking, selectingsub-regions, etc.) 3D volumetric models. For example, methods andapparatuses for displaying images from 3D volumetric models areprovided, including methods for generating sections though the 3Dvolumetric model, methods for showing both surface and internalstructures, and methods for generating easy to interpret images from the3D volumetric models, such as pseudo-x-ray images.

Also described herein are methods and apparatuses for marking andtracking regions of interest from a 3D volumetric model of a patient'sdental arch. These methods may include automatically, manually orsemi-automatically (e.g., with user approval or input) identifying oneor more regions from within the 3D volumetric model to mark (includingsurface features and/or internal features of the dental arch); theseregions may be regions in which a caries, crack or other irregularityhas developed or may develop. Marked regions may be analyzed in greaterdetail, and may be tracked over time. Further, marked regions may modifythe manner in which subsequent scanning is performed, e.g., by scanningmarked regions at higher resolution. The regions of the volumetric modelmay correspond to one or more voxels, including contiguous voxelregions. These regions may be referred to herein as volumetric regions.

Also described herein are methods and apparatuses for using 3Dvolumetric models to improve or modify a dental procedure, includingmodifying treatment planning and/or modifying one or more dental device.For example, described herein are dental tools that include 3Dvolumetric scanning, or that may be operated in conjunction with 3Dvolumetric models (including robotic or automated control using 3Dvolumetric models). Methods of diagnosing one or more conditions (e.g.,dental conditions) using a 3D volumetric model, and particularly using3D volumetric models over time are also described.

A method of displaying images from a three-dimensional (3D) volumetricmodel of a patient's dental arch, the method comprising: collecting the3D volumetric model of the patient's dental arch, wherein the 3Dvolumetric model includes surface color and shade values andnear-infrared (near-IR) transparency values for internal structureswithin the dental arch; selecting, by a user, an orientation of a viewof the 3D volumetric model to display; generating a two-dimensional (2D)view into the 3D volumetric using the selected orientation, includingthe patient's dental arch including a weighted portion of the surfacecolor values and a weighted portion of the near-IR transparency of theinternal structures; and displaying the 2D view.

For example, described herein are methods of displaying images from athree-dimensional (3D) volumetric model of a patient's dental arch. Themethod may include: receiving the 3D volumetric model of the patient'sdental arch, wherein the 3D volumetric model includes surface colorvalues and near-infrared (near-IR) transparency values for internalstructures within the dental arch; generating a two-dimensional (2D)view through the 3D volumetric model including the patient's dental archincluding both surface color values and the near-IR transparency of theinternal structures. In any of the methods and apparatuses describedherein, a 3D model (including a volumetric 3D model) may be displayed asa voxel view. Thus, the methods described herein may generate one ormore voxel views in which each voxel may have a color (or hue) thatcorresponds to its density and/or translucently. Thus, an of the methodsand apparatuses described herein may generate a 3D color map of all orsome of the voxels of the 3D model (and display one or more 2D imagesderived from the 3D color view, such a sections, slices, projections,perspective views, transparent-views in which all or some of the 3Dmodel is rendered transparent, etc.). In some variations, flaggedregions (e.g., regions corresponding to one or more irregular regions,and/or regions, e.g., voxels that have changed over time, regions/voxelsthat should be removed, regions/voxels suspected to be problematic andetc., may be displayed as a 3D and/or 2D view.

Generating the two-dimensional (2D) view through the 3D volumetric mayinclude: including in the 2D view, a weighted portion of the surfacecolor values and a weighted portion of the near-IR transparency of theinternal structures. Note that the near-IR transparency may be based onor otherwise calculated from near IR scattering or absorption of thematerial. The weighted portion of the surface color values may comprisea percentage of the full value of the surface color values, and theweighted portion of the near-IR transparency of the internal structurescomprises a percentage of the full value of the near-IR transparency ofthe internal structures, wherein the percentage of the full value of thesurface color values and the percentage of the full value of the near-IRtransparency of the internal structures adds up to 100%.

In some variations, the method also includes adjusting, by a user, or inresponse to user input, the weighted portion of the surface color valuesand/or the near-IR transparency of the internal structures.

Any of these methods may include the step of scanning the patient'sdental arch with an intraoral scanner.

Generating the 2D view may comprise sectioning the 3D volumetric modelin a plane through the 3D volumetric model. The user may select asection though the 3D volumetric model to display, and/or an orientationof the 2D view.

For example, a method of displaying images from a three-dimensional (3D)volumetric model of a patient's dental arch may include: receiving the3D volumetric model of the patient's dental arch, wherein the 3Dvolumetric model includes surface color values and near-infrared(near-IR) transparency values for internal structures within the dentalarch; selecting, by a user or in response to user input, a sectionthough the 3D volumetric model to display; generating a two-dimensional(2D) view through the 3D volumetric using the selected section,including the patient's dental arch, and possibly also including aweighted portion of the surface color values and a weighted portion ofthe near-IR transparency of the internal structures; and displaying the2D view.

A method of displaying images from a three-dimensional (3D) volumetricmodel of a patient's dental arch may include: collecting the 3Dvolumetric model of the patient's dental arch, wherein the 3D volumetricmodel includes surface values and near-infrared (near-IR) transparencyvalues for internal structures within the dental arch; generating atwo-dimensional (2D) view into the 3D volumetric model including thepatient's dental arch including both surface values and the near-IRtransparency of the internal structures; and displaying the 2D view.

A method of tracking a region of a patient's dental arch over time mayinclude: receiving a first three-dimensional (3D) volumetric model ofthe patient's dental arch, wherein the 3D volumetric model includessurface color values and near-infrared (near-IR) transparency values forinternal structures within the dental arch; identifying a region withinthe 3D volumetric model to be marked; flagging the identified region;and displaying one or more images of the 3D volumetric model indicatingthe marked region.

For example, a method of tracking a region of a patient's dental archover time, the method comprising: collecting a first three-dimensional(3D) volumetric model of the patient's dental arch, wherein the 3Dvolumetric model includes surface values and near-infrared (near-IR)transparency values for internal structures within the dental arch;identifying a region of the 3D volumetric model; flagging the identifiedregion; collecting a second 3D volumetric model of the patient's dentalarch; and displaying one or more images marking, on the one or moreimages, a difference between the first 3D volumetric model and thesecond 3D volumetric model at the flagged region.

Identifying the region may comprise automatically identifying using aprocessor. For example, automatically identifying may compriseidentifying a region having a possible defects including: cracks andcaries. Identifying the region having a possible defect may comprisecomparing a near-IR transparency value of a region within the 3D modelto a threshold value. Automatically identifying may comprise identifyinga surface color value outside of a threshold range. Automaticallyidentifying may comprise segmenting the 3D volumetric model to identifyenamel regions and identifying regions having enamel thicknesses below athreshold value. Flagging the identified region may compriseautomatically flagging the identified regions. Flagging the identifiedregion may comprise manually confirming the identified region forflagging.

Any of these methods may include receiving a second 3D volumetric modelof the patient's dental arch and displaying a difference between thefirst 3D volumetric model and the second 3D volumetric model at themarked region.

Further, any of these methods may include pre-scanning or re-scanningthe patient's dental arch wherein the flagged region is scanned at ahigher resolution or in other scanning modalities than un-flaggedregions.

For example, a method of tracking a region of a patient's dental archover time may include: receiving a first three-dimensional (3D)volumetric model of the patient's dental arch, wherein the 3D volumetricmodel includes surface color values and near-infrared (near-IR)transparency values for internal structures within the dental arch;identifying, using an automatic process, a region within the 3Dvolumetric model to be marked; flagging the identified regions;receiving a second 3D volumetric model of the patient's dental arch; anddisplaying a difference between the first 3D volumetric model and thesecond 3D volumetric model at the marked region. In some instances, thesecond 3D volumetric model of the patient's dental arch may be from ascan of the patient at a subsequent visit to the dental practitioner'soffice at a later date.

Thus, a method of tracking a region of a patient's dental arch over timemay include: collecting a first three-dimensional (3D) volumetric modelof the patient's dental arch taken at a first time, wherein the 3Dvolumetric model includes surface color values and near-infrared(near-IR) transparency values for internal structures within the dentalarch; identifying, using an automatic process, a region within the 3Dvolumetric model to be flagged; flagging the identified regions;collecting a second 3D volumetric model of the patient's dental archtaken at a separate time; and displaying a difference between the first3D volumetric model and the second 3D volumetric model at the flaggedregion.

Also described herein are methods of displaying pseudo x-ray images froma three-dimensional (3D) volumetric model of a patient's dental arch.For example, a method may include: receiving the 3D volumetric model ofthe patient's dental arch, wherein the 3D volumetric model includesnear-infrared (near-IR) transparency values for internal structureswithin the dental arch; generating a two-dimensional (2D) view throughthe 3D volumetric including the patient's dental arch including thenear-IR transparency of the internal structures; mapping the near-IRtransparency of the internal structures in the 2D view to a pseudo-X-raydensity in which the near-IR transparency values are inverted in value;and displaying the mapped pseudo-X-ray density. Generating the 2D viewmay comprise sectioning the 3D volumetric model in a plane through the3D volumetric model. The 3D volumetric model may include surfaceinformation.

For example, a method of displaying pseudo x-ray images from athree-dimensional (3D) volumetric model of a patient's dental arch mayinclude: collecting the 3D volumetric model of the patient's dentalarch, wherein the 3D volumetric model includes near-infrared (near-IR)transparency values for internal structures within the dental arch;generating a two-dimensional (2D) view into the 3D volumetric modelincluding the patient's dental arch including the near-IR transparencyof the internal structures; mapping the near-IR transparency of theinternal structures in the 2D view to a pseudo-X-ray density in whichthe pseudo-X-ray density values in the 2D view are based on the near-IRtransparency values that are inverted in value; and displaying themapped pseudo-X-ray density.

Any of these methods may include identifying a sub-region from the 3Dvolumetric model prior to generating the 2D view, wherein the 2D viewcomprises a 2D view of the identified sub-region. The method may alsoinclude segmenting the 3D volumetric model into a plurality of teeth,wherein generating the 2D view may comprise a 2D view including just oneof the identified teeth.

Mapping the near-IR transparency may include inverting the near-IRtransparency values so that enamel within the 2D view is brighter thandentin within the 2D view.

A method of displaying pseudo x-ray images from a three-dimensional (3D)volumetric model of a patient's dental arch may include: receiving the3D volumetric model of the patient's dental arch, wherein the 3Dvolumetric model includes surface features and near-infrared (near-IR)transparency values for internal structures within the dental arch inwhich enamel is more transparent than dentin; generating atwo-dimensional (2D) view through the 3D volumetric including thepatient's dental arch including the near-IR transparency of the internalstructures including dentin and enamel; mapping the near-IR transparencyof the internal structures in the 2D view to a pseudo-X-ray density inwhich the near-IR transparency values are inverted in value so that theenamel is brighter than the dentin; and displaying the mappedpseudo-X-ray density.

For example, a method of displaying pseudo x-ray images from athree-dimensional (3D) volumetric model of a patient's dental arch mayinclude: collecting the 3D volumetric model of the patient's dentalarch, wherein the 3D volumetric model includes surface features andnear-infrared (near-IR) transparency values for internal structureswithin the dental arch in which enamel is more transparent than dentin;generating a two-dimensional (2D) view into the 3D volumetric includingthe patient's dental arch including the near-IR transparency of theinternal structures including dentin and enamel; mapping the near-IRtransparency of the internal structures in the 2D view to a pseudo-X-raydensity in which the near-IR transparency values are inverted in valueso that the enamel is brighter than the dentin; and displaying themapped pseudo-X-ray density.

Also described herein are methods and apparatuses for virtuallyreviewing (e.g., virtually sectioning, virtually scanning, virtuallyexamining), in real time, a volumetric model of the patient's dentalarch(s). These apparatuses may include non-transitory, machine-readabletangible medium storing instructions for causing one or more machines toexecute operations for performing any of the methods described herein.In particular, any of these methods and apparatuses may operate on adata set that includes both a 3D model of the patient's dental arch, orin some variations, both of the patient's dental arches. The 3D modelmay be, but is not limited to, a 3D volumetric model; in some variationthe 3D model is a 3D surface model of the arch. This data set may alsoinclude a plurality of images of the dental arch, taken from differentpositions relative to the dental arch, such as different angles betweenthe plane of the image and the dental arch and different sub-regions ofthe dental arch. Some of these images may be taken from the occlusalsurface, some from the gingival side, and some from the lingual side. Insome variations the images may be the same (or a subset of) the imagesused to form the 3D model of the teeth. The data set may includemultiple images taken from the same, or nearly the same, region of thedental arch and angle relative to the dental arch. In some variations,the data set may include sets of two or more images (e.g., pairs ofimages) each taken at approximately the same region of the dental archand at the same angle relative to the dental arch but using differentimaging techniques (e.g., different imaging techniques, such as visiblelight, IR/near-IR, florescence, X-ray, ultrasound, etc.).

For example, a method may include: displaying a three-dimensional (3D)model of a patient's dental arch; displaying a viewing window over atleast a portion of the 3D model of the patient's dental arch; allowing auser to change a relative position between the viewing window and the 3Dmodel of the patient's dental arch; and continuously, as the userchanges the relative positions between the viewing window and the 3Dmodel of the patient's dental arch: identifying, from both the 3D modelof the patient's dental arch and a plurality of images of a patient'sdental arch taken from different angles and positions relative to thepatient's dental arch, an image taken at an angle and position thatapproximates a relative angle and position between the viewing windowrelative and the 3D model of the patient's dental arch; and displayingthe identified image taken at the angle and position that approximatesthe angle and position between the viewing window relative to the 3Dmodel of the patient's dental arch.

Any of the methods described herein, a data set may include the 3D modelof the patient's dental arch and a plurality of images of a patient'sdental arch taken from different angles and positions relative to thepatient's dental arch. A data set may also or alternatively includesmetadata associated with each (or each set) of the figures indicatingthe angle and/or region of the dental arch at which the image was taken.Additional metadata may be included (e.g., indicating a distance fromthe dental arch, indicating exposure time, indicating that the image isan average of other images, a quality metric for the image, etc.).

For example, described herein are methods for displaying a 3D model(e.g., surface 3D model) of the patient's teeth and/or volumetric modelof the patient's teeth) that a user can virtually scan in greater detailby moving a viewing window over the 3D model of the dental arch. Forexample, described herein are methods including: displaying athree-dimensional (3D) model of a patient's dental arch; displaying aviewing window over a portion of the 3D model of the patient's dentalarch; allowing a user to change a relative position between the viewingwindow and the 3D model of the patient's dental arch, including one ormore of: an angle between a plane of the viewing window and thepatient's dental arch, and a portion of the dental arch adjacent to theviewing window; and continuously, as the user changes the relativepositions between the viewing window and the 3D model of the patient'sdental arch: identifying, from both the 3D model of the patient's dentalarch and a plurality of images of a patient's dental arch (e.g., in somevariations from a data set comprising both the 3D model of the patient'sdental arch and a plurality of images of a patient's dental arch),wherein each image is taken from a different angle and position relativeto the patient's dental arch, an image taken at an angle and positionthat approximates the relative angle and position between the viewingwindow relative and the 3D model of the patient's dental arch; anddisplaying the identified image taken at the angle and position thatapproximates the angle and position of the viewing window relative tothe displayed 3D model of the patient's dental arch.

For example, a method may include: displaying a three-dimensional (3D)model of a patient's dental arch; displaying a viewing window over aportion of the 3D model of the patient's dental arch; allowing a user tochange a relative position between the viewing window and the 3D modelof the patient's dental arch, including one or more of: an angle betweenthe patient's dental arch relative and a plane of the viewing window,and a portion of the dental arch adjacent to the viewing window; andcontinuously, as the user changes the relative position between theviewing window and the 3D model of the patient's dental arch:identifying, from both the 3D model of the patient's dental arch and aplurality of pairs of images of a patient's dental arch (e.g.,optionally from a data set comprising both the 3D model of the patient'sdental arch and a plurality of images of a patient's dental arch),wherein each pair of the plurality of pairs includes a first imagingwavelength and a second imaging wavelength each taken at the same angleand position relative to the patient's dental arch, a pair of imagestaken at an angle and position that approximate the angle and positionof the viewing window relative to the displayed 3D model of thepatient's dental arch; and displaying at least one of the identifiedpair of images taken at the angle and position that approximate theangle and position of the viewing window relative to the displayed 3Dmodel of the patient's dental arch.

The methods and apparatuses described herein can be used with a 3D modelthat is a surface model or any representation of the patient's dentalarch(s). It may be, but does not have to be, a 3D volumetric model ofthe patient's teeth, e.g., constructed from images (e.g., the pluralityof images of a patient's dental arch taken from different angles andpositions relative to the patient's dental arch). The model may berepresentative of the patient's actual dentition, abstracted from thepatient's dentition, or generic.

As described herein, a method may include: displaying athree-dimensional (3D) model of a patient's dental arch; displaying aviewing window over a portion of the 3D model of the patient's dentalarch; allowing a user to change a relative position between the viewingwindow and the 3D model of the patient's dental arch, including one ormore of: an angle between the viewing window and the patient's dentalarch, and a portion of the dental arch adjacent to the viewing window;and continuously, as the user changes the relative positions between theviewing window and the 3D model of the patient's dental arch:identifying, both the 3D model of the patient's dental arch and aplurality of near-IR images of a patient's dental arch (e.g., from adata set comprising both the 3D model of the patient's dental arch and aplurality of images of a patient's dental arch), wherein each near-IRimage is taken from a different angle and position relative to thepatient's dental arch, a near-IR image taken at an angle and positionthat approximates the relative angle and position between the viewingwindow relative and the 3D model of the patient's dental arch; anddisplaying the identified near-IR image taken at the angle and positionthat approximates the angle and position of the viewing window relativeto the displayed 3D model of the patient's dental arch.

In any of these examples, the images may be images taken with apenetrating modality, such as with a near-IR. For example, describedherein are methods including: displaying a three-dimensional (3D) modelof a patient's dental arch; displaying a viewing window over a portionof the 3D model of the patient's dental arch; allowing a user to changea relative position between the viewing window and the 3D model of thepatient's dental arch, including one or more of: an angle between theviewing window and the patient's dental arch, and a portion of thedental arch adjacent to the viewing window; and continuously, as theuser changes the relative positions between the viewing window and the3D model of the patient's dental arch: identifying, from a data setcomprising both the 3D model of the patient's dental arch and aplurality of near-IR images of a patient's dental arch, wherein eachnear-IR image is taken from a different angle and position relative tothe patient's dental arch, a near-IR image taken at an angle andposition that approximates the relative angle and position between theviewing window relative and the 3D model of the patient's dental arch;and displaying the identified near-IR image taken at the angle andposition that approximates the angle and position of the viewing windowrelative to the displayed 3D model of the patient's dental arch.

Any of these methods may also include identifying and displayingmultiple images taken at the same angle and position relative to thedental arch. For example, the images may be both a visible light imageand a penetrative image (such as an IR/near-IR image, etc.). Forexample, described herein are: methods comprising: displaying athree-dimensional (3D) model of a patient's dental arch; displaying aviewing window over a portion of the 3D model of the patient's dentalarch; allowing a user to change a relative position between the viewingwindow and the 3D model of the patient's dental arch, including one ormore of: an angle between the patient's dental arch relative and a planeof the viewing window, and a portion of the dental arch adjacent to theviewing window; and continuously, as the user changes the relativeposition between the viewing window and the 3D model of the patient'sdental arch: identifying, from a data set comprising both the 3D modelof the patient's dental arch and a plurality of pairs of images of apatient's dental arch, wherein each pair of the plurality of pairsincludes a first imaging wavelength and a second imaging wavelength eachtaken at the same angle and position relative to the patient's dentalarch, a pair of images taken at an angle and position that approximatethe angle and position of the viewing window relative to the displayed3D model of the patient's dental arch; and displaying the identifiedpair of images taken at the angle and position that approximate theangle and position of the viewing window relative to the displayed 3Dmodel of the patient's dental arch.

In any of these methods, identifying may comprise determining aplurality images that approximate the relative angle and positionbetween the viewing window relative and the 3D model of the patient'sdental arch and averaging the plurality to form the identified image.For example, there may be multiple images in the data set taken atapproximately (e.g., within +/−0.1%, 0.5%, 1%, 2%, 3%, 4%, 5%, 7%, 10%,15%, 20%, etc.) of the same angle and approximately (e.g., within+/−0.1%, 0.5%, 1%, 2%, 3%, 4%, 5%, 7%, 10%, 15%, 20%, etc.) of the sameregion of the dental arch; these similar images may be combined to forman average image that may be better than the individual images.

In general, identifying one or more images taken at an angle andposition that approximates the relative angle and position between theviewing window relative and the 3D model of the patient's dental archmay be identifying within an acceptable spatial range. For example, animage that was taken at between +/−a few degrees of the same angle(e.g., +/−0.1 degree, 0.2 degree, 0.3 degrees, 0.4 degrees, 0.5 degrees,0.6 degrees, 1 degree, 1.2 degrees, 1.5 degrees, 1.7 degrees, 1.8degrees, 2 degrees, 2.2 degrees, 2.5 degrees, 3 degrees, 3.2 degrees,3.5 degrees, 4 degrees, 5 degrees, etc.) as the plane of the viewingwidow and within +/−a range of distance of the dental arch region overwhich the viewing window is positioned (e.g., +/−0.1 mm, 0.2 mm, 0.3 mm,0.4 mm, 0.5 mm, 0.6 mm, 0.7 mm, 0.8 mm, 0.9 mm, 1 mm, 1.1 mm, 1.2 mm,1.5 mm, 1.7 mm, 2.0 mm, 2.2 mm, 2.5 mm, etc.).

Any of these methods may include receiving, in a processor, the dataset. The data set may be received directly from an intraoral scanner,and/or stored and retrieved. In some variations the data set may betransmitted and received by the processor, in some variations theprocessor may read the data set from a memory (e.g., a data store)connected to the processor.

In general, any of these methods may include displaying the viewingwindow over a portion of the 3D model of the patient's dental arch. Theviewing window may be any shape or size, such as a circle, oval,triangle, rectangle, or other polygon. For example, the viewing windowmay be a loop through which the portion of the 3D model of the patient'sdental arch may be viewed. The viewing angle may allow the dental archto be visualized through at least a portion of the viewing window. Theviewing window may be smaller than the dental arch. In some variationsthe viewing window may be made larger or smaller by the user.

Typically these methods may include displaying via a user interface. Forexample, the user interface may display on a screen or screens thedental arch 3D model, the viewing window, and/or the image(s)corresponding to the view thorough the viewing window of the dentalarch. The user may (e.g., by manipulating the user interface, e.g., viaa control such as a mouse, keyboard, touchscreen, etc.) move the viewingwindow and dental arch independently. This movement, and the image(s)determined to correspond to the image though the viewing window of theregion and angle of the viewing window relative to the dental arch, maybe displayed in real time, as the user moves the viewing window and/ordental arch relative to each other.

For example, allowing the user to change the relative position betweenthe viewing window and the 3D model of the patient's dental arch mayinclude separately controlling the angle and/or rotation of the 3D modelof a patient's dental arch and the portion of the dental arch adjacentto the viewing window. In some variations, allowing the user to changethe relative position between the viewing window and the 3D model of thepatient's dental arch may comprise allowing the user to move the viewingwindow over the 3D model of the dental arch.

As mentioned, any of the images identified to as taken from an angle andposition corresponding to the angle and position of the viewing windowas it is moved over and/or around the dental arch (or as the dental archis moved relative to the viewing window) may be any one or moremodalities. Thus, for example, identifying an image that approximatesthe relative angle and position between the viewing window relative andthe 3D model of the patient's dental arch may include identifying oneof: a visible light image, an infrared image, and a florescent image.

Displaying the identified image(s) that approximates the angle andposition of the viewing window relative to the displayed 3D model maycomprise displaying the identified image in a window adjacent oroverlapping with the display of the 3D model of the patient's dentalarch. For example, the images may be displayed on a screen alongside the3D model of the dental arch; a the user moves the dental arch and/orimaging window, the image(s) may be shown in one or more windowschanging in real time or near real-time to reflect the relative positionof the 3D model of the dental arch and the viewing window.

Also described herein are non-transitory, machine-readable tangiblemedium storing instructions for causing one or more machines to executeoperations for performing any of the methods described herein, includingvirtually reviewing a patient's dental arch. For example, anon-transitory, machine-readable tangible medium may store instructionsfor causing one or more machines to execute operations for virtuallyreviewing a patient's dental arch including: displaying athree-dimensional (3D) model of a patient's dental arch; displaying aviewing window over a portion of the 3D model of the patient's dentalarch; allowing a user to change a relative position between the viewingwindow and the 3D model of the patient's dental arch, including one ormore of: an angle between the viewing window and the patient's dentalarch, and a portion of the dental arch adjacent to the viewing window;and continuously, as the user changes the relative positions between theviewing window and the 3D model of the patient's dental arch:identifying, from a data set comprising both the 3D model of thepatient's dental arch and a plurality of images of a patient's dentalarch, wherein each image is taken from a different angle and positionrelative to the patient's dental arch, an image taken at an angle andposition that approximates the relative angle and position between theviewing window relative and the 3D model of the patient's dental arch;and displaying the identified image taken at the angle and position thatapproximates the angle and position of the viewing window relative tothe displayed 3D model of the patient's dental arch.

For example, a non-transitory, machine-readable tangible medium storinginstructions for causing one or more machines to execute operations forvirtually reviewing a patient's dental arch, comprising: displaying athree-dimensional (3D) model of a patient's dental arch; displaying aviewing window over a portion of the 3D model of the patient's dentalarch; allowing a user to change a relative position between the viewingwindow and the 3D model of the patient's dental arch, including one ormore of: an angle between the viewing window and the patient's dentalarch, and a portion of the dental arch adjacent to the viewing window;and continuously, as the user changes the relative positions between theviewing window and the 3D model of the patient's dental arch:identifying, from a data set comprising both the 3D model of thepatient's dental arch and a plurality of images of a patient's dentalarch, wherein each image is taken from a different angle and positionrelative to the patient's dental arch, an image taken at an angle andposition that approximates the relative angle and position between theviewing window relative and the 3D model of the patient's dental arch;and displaying the identified image taken at the angle and position thatapproximates the angle and position of the viewing window relative tothe displayed 3D model of the patient's dental arch.

Also described herein are intraoral scanning systems that are configuredto perform the methods described herein. For example, an intraoralscanning system may include a hand-held wand having at least one imagesensor and a light source configured to emit light at a spectral rangewithin near-infrared (near-IR) range of wavelengths; a display output(e.g., a visual output such as a monitor, screen, virtual realityinterface/augmented reality interface, etc.); a user input device (e.g.,any control for receiving and transmitting user input, such as, but notlimited to: a keyboard, button, joystick, touchscreen, etc. The displayoutput and the user input device may be the same touchscreen); and oneor more processors operably connected to the hand-held wand, display anduser input device, the one or more processors configured to: display athree-dimensional (3D) model of a patient's dental arch on the displayoutput; display a viewing window over a portion of the 3D model of thepatient's dental arch on the display output; change a relative positionbetween the viewing window and the 3D model of the patient's dental archbased on input from the user input device; identify, from both the 3Dmodel of the patient's dental arch and a plurality of images of thepatient's dental arch taken from different angles and positions relativeto the patient's dental arch, a near-infrared (near-IR) image taken atan angle and position that approximates a relative angle and positionbetween the viewing window relative and the 3D model of the patient'sdental arch; and display the identified near-IR image taken at the angleand position that approximates the angle and position between theviewing window relative to the 3D model of the patient's dental arch.

The one or more processors of the intraoral scanning system may beconfigured to receive the plurality of images of the patient's dentalarch taken from different angles and positions relative to the patient'sdental arch. For example, the images may be taken by the image sensor(s)on the hand-held wand and transmitted to the one or more processorsand/or stored in a memory that is accessed by the one or moreprocessors. The system may also include a controller coordinating theactivity of the one or more processors, the wand, and the display output(and user input device). The controller may display the images and/or a3D model constructed from the images as a user operates the hand-heldwant to take images at different locations and/or angles relative to thepatient's dental arch(es).

The one or more processors may be configured to continuously identifythe near-IR image and display the identified near-IR image as the userchanges the relative positions between the viewing window and the 3Dmodel of the patient's dental arch. Thus, as the user (using the userinput) adjusts the position of the viewing window (e.g., loop) relativeto the 3D model of the patient's dental arch on the display output (or,equivalently, adjusts the position of the 3D model of the dental arch onthe display output relative to the viewing window), the one or moreprocessors may determine and display a near-IR image of the patient'steeth that most closely approximates the relative positions between theviewing window and the 3D model of the patient's dental arch.

The near-IR image is either one of the images taken by the hand-heldwand or an average of the images taken by the hand-held wand. Any of theapparatuses (e.g., intraoral scanning systems) described herein may alsodetermine and/or store the positions and/or orientation of the hand-heldwand as it is being operated, and this information may be stored withthe image(s) taken from this position. For example, the hand-held wandmay include one or more accelerometers. For example, the one or moreprocessors may be configured to identify the near-IR image taken at anangle and position that approximates a relative angle and positionbetween the viewing window relative and the 3D model of the patient'sdental arch by determining a plurality images that approximate therelative angle and position between the viewing window relative and the3D model of the patient's dental arch and averaging the plurality toform the identified near-IR image.

As mentioned, the one or more processors may be configured to change, onthe display output, the relative position between the viewing window andthe 3D model of the patient's dental arch based on input from the userinput device. Specifically, the one or more processor may be configuredto change, based on user input into user input device, one or more of:an angle between a plane of the viewing window and the patient's dentalarch, and a portion of the dental arch adjacent to the viewing window(e.g., in some variations, visible through the viewing window). Asdiscussed above, the viewing window may be a loop (e.g., circular, oval,square, etc.) through which the 3D model is visible). Thus, the one ormore processors may be configured to display the viewing window over aportion of the 3D model of the patient's dental arch comprisesdisplaying as a loop through which the portion of the 3D model of thepatient's dental arch may be viewed. The viewing window may be moved andpositioned over (including changing which side of the dental arch(buccal, occlusal, lingual, or between these, including moving in x, y,z and/or in rotation, e.g., pitch, roll, yaw) the viewing window ispositioned over and/or the 3D model of the patient's teeth may be moved(e.g., rotating in pitch, yaw, roll, moving in x, y, z, etc.). Thus, theone or more processors may be configured to change the relative positionbetween the viewing window and the 3D model of the patient's dental archbased on input from the user input device by changing one or more of:the angle of the 3D model of a patient's dental arch relative to theviewing window (which is equivalent to the angle of the viewing windowrelative to the 3D model of the patient's dental arch), the rotation ofthe 3D model of a patient's dental arch relative to the viewing window(which is equivalent to the rotation of the viewing window relative tothe 3D model of a patient's dental arch), and the portion of the dentalarch adjacent to the viewing window (e.g., the portion of the 3D modelvisible through the viewing window). For example, the one or moreprocessors may be configured to change the relative position between theviewing window and the 3D model of the patient's dental arch based oninput from the user input device by changing the position of the viewingwindow over the 3D model of the dental arch.

The one or more processors may be configured to identify from both the3D model of the patient's dental arch and the plurality of images of thepatient's dental arch taken from different angles and positions relativeto the patient's dental arch, a second image that approximates therelative angle and position between the viewing window relative and the3D model of the patient's dental arch that is one or more of: a visiblelight image and a florescent image; and wherein the one or moreprocessors is configured to display the second image concurrently withthe near-IR image.

Also described herein are methods of automatically,semi-automatically/semi-manually or manually identifying and gradingfeatures by coordinating across multiple imaging modalities. Forexample, a dental diagnostic method may include: identifying a dentalfeature in a first record, the first record comprising a plurality ofimages of a patient's dental arch taken first imaging modality;correlating the first record with a model of the patient's dental arch;identifying, using the model of the patient's dental arch, a region ofthe dental arch corresponding to the dental feature in one or moredifferent records, wherein each record of the one or more differentrecords is taken with a different imaging modality than the firstimaging modality and wherein each of the one or more different recordsis correlated with the model of the patient's dental arch; determining aconfidence score for the dental feature based on the identified regionscorresponding to the dental feature in the one or more differentrecords; and displaying the dental feature when the confidence score forthe dental feature is above a threshold.

A dental diagnostic method may include: identifying a dental feature ina first record, the first record comprising a plurality of images of apatient's dental arch taken first imaging modality; correlating thefirst record with a three-dimensional (3D) volumetric model of thepatient's dental arch; flagging the dental feature on the 3D volumetricmodel; identifying, using the model of the patient's dental arch, aregion of the dental arch corresponding to the dental feature in one ormore different records, wherein each record of the one or more differentrecords is taken with a different imaging modality than the firstimaging modality and wherein each of the one or more different recordsis correlated with the model of the patient's dental arch; determiningor adjusting a confidence score for the dental feature based on theidentified regions corresponding to the dental feature in the one ormore different records; and displaying the dental feature and anindicator of the confidence score for the dental feature when theconfidence score for the dental feature is above a threshold.

In any of these methods (or systems for performing them) the dentalfeature may comprise one or more of: cracks, gum recess, tartar, enamelthickness, pits, caries, pits, fissures, evidence of grinding, andinterproximal voids.

Displaying may comprise displaying the dental feature and an indicatorof the confidence score for the dental feature.

Correlating the first record with the model of the patient's dental archmay comprise correlating the first record with a three-dimensional (3D)volumetric model of the patient's dental arch. Any of these methods (orsystems for performing them) may include flagging the dental feature onthe model of the patient's dental arch, and/or collecting the dentalfeature, including the location of the dental feature, and one or moreof: the type of dental feature and a confidence score for the dentalfeature.

Determining the confidence score may comprise adjusting the confidencescore for the dental feature based on the identified regionscorresponding to the dental feature in the one or more differentrecords.

In any of these methods or systems, identifying the dental feature maycomprise automatically identifying the dental feature.

For example, a dental diagnostic method may include: identifying one ormore actionable dental features from one or more records of a pluralityof records, wherein each record comprises a plurality of images of apatient's dental arch each taken using an imaging modality, furtherwherein each record of the plurality of records is taken at a differentimaging modality; mapping the actionable dental feature to acorresponding region of the one or more records; recording the one ormore actionable dental features, including recording a location of theactionable dental feature; adjusting or determining a confidence scorefor the one or more actionable dental features based on thecorresponding region of the one or more records; and displaying the oneor more actionable dental features when the confidence score of the oneor more actionable dental features is above a threshold. As mentionedabove, the one or more actionable dental feature comprises one or moreof: cracks, gum recess, tartar, enamel thickness, pits, caries, pits,fissures, evidence of grinding, and interproximal voids.

Displaying may comprise displaying the one or more actionable dentalfeatures and an indicator of the confidence score for the dentalfeature. Mapping the actionable dental feature to the correspondingregion of the one or more records may comprise correlating the firstrecord with a three-dimensional (3D) volumetric model of the patient'sdental arch. Recording the one or more actionable dental features maycomprise marking the dental feature on the 3D volumetric model of thepatient's dental arch. Identifying the dental feature may compriseautomatically identifying the dental feature.

Also described herein are systems for performing any of the methodsdescribed herein. For example, a system may include: one or moreprocessors; and a memory coupled to the one or more processors, thememory configured to store computer-program instructions, that, whenexecuted by the one or more processors, perform a computer-implementedmethod comprising: identifying a dental feature in a first record, thefirst record comprising a plurality of images of a patient's dental archtaken first imaging modality; correlating the first record with a modelof the patient's dental arch; identifying, using the model of thepatient's dental arch, a region of the dental arch corresponding to thedental feature in one or more different records, wherein each record ofthe one or more different records is taken with a different imagingmodality than the first imaging modality and wherein each of the one ormore different records is correlated with the model of the patient'sdental arch; determining a confidence score for the dental feature basedon the identified regions corresponding to the dental feature in the oneor more different records; and displaying the dental feature when theconfidence score for the dental feature is above a threshold.

Also described herein are methods and apparatuses (e.g., systems) fortracking one or more regions (e.g., tagged or flagged regions) acrossdifferent imaging modalities and/or over time. For example, a method oftracking a dental feature across different imaging modalities mayinclude: collecting a first three-dimensional (3D) volumetric model ofthe patient's dental arch, wherein the 3D volumetric model of thepatient's dental arch includes surface values and internal structureswithin the dental arch; identifying a region of the patient's dentalarch from a first record of a plurality of records, wherein each recordcomprises a plurality of images of a patient's dental arch each takenusing an imaging modality, further wherein each record of the pluralityof records is taken at a different imaging modality; flagging theidentified region in a corresponding region of the 3D volumetric modelof the patient's dental arch; correlating the flagged region with eachof records of the plurality of records by correlating the 3D volumetricmodel of the patient's dental arch with each of the records of theplurality of records; and saving, displaying and/or transmitting imagesincluding the region of the patient's dental arch. The region of thepatient's dental arch may comprise a dental feature comprises one ormore of: cracks, gum recess, tartar, enamel thickness, pits, caries,pits, fissures, evidence of grinding, and interproximal voids.

Saving, displaying and/or transmitting may comprise displaying theregions of the patient's dental arch. Any of these methods may includeflagging the dental feature on the 3D volumetric model. Identifying theregion of the patient's dental arch may comprise automaticallyidentifying the region of the patient's dental arch.

A system for tracking one or more regions (e.g., tagged or flaggedregions) across different imaging modalities and/or over time mayinclude: one or more processors; a memory coupled to the one or moreprocessors, the memory configured to store computer-programinstructions, that, when executed by the one or more processors, performa computer-implemented method comprising: collecting a firstthree-dimensional (3D) volumetric model of the patient's dental arch,wherein the 3D volumetric model of the patient's dental arch includessurface values and internal structures within the dental arch;identifying a region of the patient's dental arch from a first record ofa plurality of records, wherein each record comprises a plurality ofimages of a patient's dental arch each taken using an imaging modality,further wherein each record of the plurality of records is taken at adifferent imaging modality; flagging the identified region in acorresponding region of the 3D volumetric model of the patient's dentalarch; correlating the flagged region with each of records of theplurality of records by correlating the 3D volumetric model of thepatient's dental arch with each of the records of the plurality ofrecords; and saving, displaying and/or transmitting images including theregion of the patient's dental arch.

BRIEF DESCRIPTION OF THE DRAWINGS

The novel features of the invention are set forth with particularity inthe claims that follow. A better understanding of the features andadvantages of the present invention will be obtained by reference to thefollowing detailed description that sets forth illustrative embodiments,in which the principles of the invention are utilized, and theaccompanying drawings of which:

FIG. 1A illustrates one example of a 3D (color) intraoral scanner thatmay be adapted for used as described herein to generate a model ofsubject's teeth having both surface and internal features.

FIG. 1B schematically illustrates an example of an intraoral scannerconfigured to generate a model of subject's teeth having both surfaceand internal features.

FIG. 2 shows a schematic of an intraoral scanner configured to do bothsurface scanning (e.g., visible light, non-penetrative) and penetrativescanning using a near infra-red (IR) wavelength. The scanner includes apolarizer and filters to block near-IR light reflected off the surfaceof the tooth while still collecting near-IR light reflected frominternal structures.

FIG. 3 is an example of a method of scanning teeth with an intraoralscanner to identify internal structures using a penetrative wavelength(e.g., IR and/or near-IR).

FIG. 4 illustrates one method of generating internal structure (orpseudo x-ray) images from a volumetric data.

FIGS. 5A and 5B illustrate virtual sections from a volumetric model ofthe teeth. These virtual sections may be annotated,colored/pseudo-colored, or textured, to show internal features orproperties of the teeth. In FIG. 5A, virtual section is pseudo-coloredto show enamel; in FIG. 5B, the virtual section is pseudo-colored toshow dentin.

FIG. 6 illustrates one method of marking (e.g., flagging) a volumetricmodel of a patient's teeth, and/or using the marked regions.

FIG. 7 is a comparison between a typical computer-aideddesign/computer-aided manufacturing (CAD/CAM) method for dentistry, anda method implementing the 3D volumetric scanning and modeling asdescribed herein.

FIG. 8A is an example of a display tracking gingival recession over timein using a 3D volumetric model as described herein. FIG. 8B shows anenlarged view of region B in FIG. 8A showing the later time.

FIGS. 9A-9G illustrate one method of displaying volumetric informationfrom a patient's teeth. FIG. 9A show an example of a 3D volumetric modelof a patient's upper jaw (showing teeth and gingiva), from a top view.FIG. 9B shows the same 3D volumetric model, showing the internalfeatures, including the more transparent enamel and the less transparentdentin. The 3D volumetric model may be manipulated to show more or lessof the surface and/or internal structures. FIGS. 9C-9G illustrateprogressively more transparent views or a region (“C”) of the 3Dvolumetric model of FIG. 9A. FIG. 9C show a 2D image extracted from aregion of the 3D volumetric model showing just the outer surface of theteeth (e.g., 100% of the color/outer surface image, 0% near-IR/internalvolume). FIG. 9D shows the same region as FIG. 9C, combining the outersurface (color) image and the internal (near-IR based) image (e.g., 75%of the color/outer surface image, 25% near-IR/internal volume). FIG. 9Eshows the same region as FIG. 9C, combining the outer surface (color)image and the internal (near-IR based) image (e.g., 50% of thecolor/outer surface image, 50% near-IR/internal volume). FIG. 9F showsthe same region as FIG. 9C, combining the outer surface (color) imageand the internal (near-IR based) image (e.g., 25% of the color/outersurface image, 75% near-IR/internal volume). FIG. 9G shows the sameregion as FIG. 9C showing just the internal (near IR based) image of theteeth (e.g., 0% of the color/outer surface image, 1000% near-IR/internalvolume).

FIG. 10A illustrates an example of a user interface for analysis and/ordisplay of a 3D volumetric model of a patient's teeth, showing a topview of the upper arch, tools that may be used to manipulate theview(s), and two enlarged views showing the outer surface of an enlargedregion of the tooth (on the left) and the same view showing internalfeatures of the tooth (showing dentin and enamel within the tooth).

FIG. 10B show the user interface of FIG. 10A in which a region of theteeth has been marked/flagged as described herein.

FIGS. 11A-11C illustrate another example of a method of displaying 3Dvolumetric image information by mixing it with surface (non-penetrative)information. FIG. 11A shows a visible light image of a region of apatient's dental arch taken with a scanner that is also configured totake penetrative (near-IR) scans). FIG. 11B show a volumetric model ofthe reconstructed 3D volumetric model of a patient's tooth showinginternal dentin and enamel. Features not visible on the surface scan areapparent in the volumetric scan, including a caries and a bubbled regionwithin the enamel. FIG. 11C shows a hybrid image in which the 3Dvolumetric image has been combined with the surface scan, showing bothsurface and internal structures, including the carries and the bubbledregion.

FIG. 12 is an example of a method for allowing a user to virtually scana patient's dental arch. This method may be performed in real time ornear real time.

FIG. 13 is a schematic illustration of a data structure including a 3Dmodel of a patient's dental arch(s) and associated 2D images taken(e.g., via intraoral scanner) of the dental arch at a large number ofpositions around the dental arch.

FIG. 14A is an example of a user interface allowing the user tovirtually scan over the 3D model of the dental arch, showingcorresponding light and near-IR (e.g., external and internal) regions indetail as the user scans over the 3D dental arch; the user may use oneor more tools to move the dental arch (e.g., rotate, translate, etc.)and/or the viewing window; the corresponding light and near-IR imagesmay continuously or near-continuously update as the position of theviewing window and dental arch change. A pair of imaging windows areshown adjacent to the view of the 3D model of the dental arch.

FIG. 14B is an alternative display showing a single large image windowover or adjacent to the 3D image of the dental arch. In FIG. 14B theimage window show a light image of the corresponding region of thedental arch.

FIG. 14C is an alternative display showing a single large image windowover or adjacent to the 3D image of the dental arch. In FIG. 14C theimage window show a near-IR image of the corresponding region of thedental arch.

FIG. 15A is similar to FIG. 14A, showing an example of a 3D model of theouter surface of a dental arch, and a viewing window relative to thedental arch. A pair of image display windows are adjacent to the 3Dmodel of the dental arch. The user may move the viewing window over thedental arch (and/or may move the dental arch relative to the viewingwindow, changing the image(s) shown in the two display windows. Theupper display window shows a near-IR image corresponding to the dentalarch at the position and angle of the plane of the viewing window; thebottom display window shows a corresponding light image (which may be incolor).

FIG. 15B shows another image of the dental arch shown in FIG. 15A, withthe dental arch rotated lingually relative to the viewing window; thecorresponding near-IR images (upper right) and visible light (lowerright) adjacent to the 3D model of the arch are updated to show theslightly rotated view, allowing the user to virtually scan the dentalarch and show both external and internal views in real (or near-real)time.

FIG. 16A is another example of a method of showing a 3D model of adental arch (shown as the lower arch in this example, e.g., by selectingthe lower arch display control in the upper left of the user interface)and showing focused views of near-IR and visible light imagescorresponding to the viewing window region that may be movedover/across, and around (lingual-occlusal-buccal) the model of thepatient's arch.

FIG. 16B shows an example of a single window (an enlarged near-IR viewinto the teeth of the region corresponding to the viewing window loop)similar to FIG. 16A.

FIG. 16C shows an example of a single window (an enlarged visible lightview into the teeth of the region corresponding to the viewing windowloop) similar to FIG. 16A.

FIG. 17 schematically illustrates one example of a method forautomatically or semi-automatically identify, confirm and/orcharacterize one or more actionable dental features that may benefitfrom detection and/or treatment.

DETAILED DESCRIPTION

Described herein are methods and apparatuses (e.g., devices and systems)that apply scans of both external and/or internal structures of teeth.These methods and apparatuses may generate and/or manipulate a model ofa subject's oral cavity (e.g., teeth, jaw, palate, gingiva, etc.) thatmay include both surface topography and internal features (e.g., dentin,dental filling materials (including bases and linings), cracks and/orcaries). Apparatuses for performing both surface and penetrativescanning of the teeth may include intraoral scanners for scanning intoor around a subject's oral cavity and that are equipped with a lightsource or light sources that can illuminate in two or more spectralranges: a surface-feature illuminating spectral range (e.g., visiblelight) and a penetrative spectral range (e.g. IR range, and particularly“near-IR,” including but not limited to 850 nm). The scanning apparatusmay also include one or more sensors for detecting the emitted light andone or more processors for controlling operation of the scanning and foranalyzing the received light from both the first spectral range and thesecond spectral range to generate a model of the subject's teethincluding the surface of the teeth and features within the teeth,including within the enamel (and/or enamel-like restorations) anddentin. The generated mode may be a 3D volumetric model or a panoramicimage.

As used herein, a volumetric model may include a virtual representationof an object in three dimensions in which internal regions (structures,etc.) are arranged within the volume in three physical dimensions inproportion and relative relation to the other internal and surfacefeatures of the object which is being modeled. For example, a volumetricrepresentation of a tooth may include the outer surface as well asinternal structures within the tooth (beneath the tooth surface)proportionately arranged relative to the tooth, so that a sectionthrough the volumetric model would substantially correspond to a sectionthrough the tooth, showing position and size of internal structures; avolumetric model may be section from any (e.g., arbitrary) direction andcorrespond to equivalent sections through the object being modeled. Avolumetric model may be electronic or physical. A physical volumetricmodel may be formed, e.g., by 3D printing, or the like. The volumetricmodels described herein may extend into the volume completely (e.g.,through the entire volume, e.g., the volume of the teeth) or partially(e.g., into the volume being modeled for some minimum depth, e.g., 2 mm,3 mm, 4 mm, 5 mm, 6 mm, 7 mm, 8 mm, 9 mm, 10 mm, 12 mm, etc.).

The methods described herein typically include methods for generating amodel of a subject's teeth typically generating a 3D model or renderingof the teeth that include both surface and internal features.Non-ionizing methods of imaging and/or detecting internal structures maybe used, such as taking images using a penetrating wavelength to viewstructures within the teeth by illuminating them using one or morepenetrative spectral ranges (wavelengths), including usingtrans-illumination (e.g., illuminating from one side and capturing lightfrom the opposite side after passing through the object), and/orsmall-angle penetration imaging (e.g., reflective imaging, capturinglight that has been reflected/scattered from internal structures whenilluminating with a penetrating wavelength). In particular, multiplepenetration images may be taken from the same relative position.Although traditional penetration imaging techniques (e.g.,trans-illumination) may be used, in which the angle between the lightemitter illumination direction and the detector (e.g., camera) viewangle is 90 degrees or 180 degrees, also described herein are methodsand apparatuses in which the angle is much smaller (e.g., between 0degrees and 25 degrees, between 0 degrees and 20 degrees, between 0degrees and 15 degrees, between 0 degrees and 10 degrees, etc.). Smallerangles (e.g., 0-15°) may be particularly beneficial because theillumination (light source) and sensing (detector(s), e.g., camera(s),etc.) may be closer to each other, and may provide a scanning wand forthe intraoral scanner that can be more easily positioned and movedaround a subject's teeth. These small-angle penetration images andimaging techniques may also be referred to herein as reflectiveillumination and/or imaging, or as reflective/scattering imaging. Ingeneral penetrating imaging may refer to any appropriate type ofpenetrating imaging unless otherwise specified, includingtrans-illumination, small-angle penetration imaging, etc. However, smallangles may also result in direct reflection from the surface of theobject (e.g., teeth), which may obscure internal structures.

The methods and apparatuses described here are particularly effective incombining a 3D surface model of the tooth or teeth with the imagedinternal features such as lesions (caries, cracks, etc.) that may bedetected by the use of penetration imaging by using an intraoral scannerthat is adapted for separate but concurrent (or nearly-concurrent)detection of both the surface and internal features. Combining surfacescanning and the penetration imaging may be performed by alternating orswitching between these different modalities in a manner that allows theuse of the same coordinate system for the two. Alternatively, bothsurface and penetrative scanning may be simultaneously viewed, forexample, by selectively filtering the wavelengths imaged to separate theIR (near-IR) light from the visible light. The 3D surface data maytherefore provide important reference and angle information for theinternal structures, and may allow the interpretation and analysis ofthe penetrating images that may otherwise be difficult or impossible tointerpret.

The penetrative scans described herein may be collected from, forexample, an intraoral scanner such as the one illustrated in FIGS. 1A-1Bfor generating a three-dimensional (3D) model of a subject's intraoralregion (e.g., tooth or teeth, gums, jaw, etc.) which may includeinternal features of the teeth and may also include a model of thesurface, and methods of using such scanners. Although in many instancessurface scanning (including color scans) may be helpful and useful, thepenetrative (IR) scanning may, in some of the variations describedherein, be sufficient.

In FIG. 1A the exemplary intraoral scanner 101 may be configured oradapted to generate 3D models having both surface and internal features,or just internal (penetrative) scans. As shown schematically in FIG. 1B,an exemplary intraoral scanner may include a handle or wand 103 that canbe hand-held by an operator (e.g., dentist, dental hygienist,technician, etc.) and moved over a subject's tooth or teeth to scan bothsurface and internal structures. The wand may include one or moresensors 105 (e.g., cameras such as CMOS, CCDs, detectors, etc.) and oneor more light sources 109, 110, 111. In FIG. 1B, three light sources areshown: a first light source 109 configured to emit light in a firstspectral range for detection of surface features (e.g., visible light,monochromatic visible light, etc.; this light does not have to bevisible light), a second color light source (e.g., white light between400-700 nm, e.g., approximately 400-600 nm), and a third light source111 configured to emit light in a second spectral range for detection ofinternal features within the tooth (e.g., by trans-illumination,small-angle penetration imaging, laser florescence, etc., which maygenerically be referred to as penetration imaging, e.g., in thenear-IR). Although separate illumination sources are shown in FIG. 1B,in some variations a selectable light source may be used. The lightsource may be any appropriate light source, including LED, fiber optic,etc. The wand 103 may include one or more controls (buttons, switching,dials, touchscreens, etc.) to aid in control (e.g., turning the wandon/of, etc.); alternatively or additionally, one or more controls, notshown, may be present on other parts of the intraoral scanner, such as afoot petal, keyboard, console, touchscreen, etc.

In general, any appropriate light source may be used, in particular,light sources matched to the mode being detected. For example, any ofthese apparatuses may include a visible light source or other (includingnon-visible) light source for surface detection (e.g., at or around 680nm, or other appropriate wavelengths). A color light source, typically avisible light source (e.g., “white light” source of light) for colorimaging may also be included. In addition a penetrating light source forpenetration imaging (e.g., infrared, such as specifically near infraredlight source) may be included as well.

The intraoral scanner 101 may also include one or more processors,including linked processors or remote processors, for both controllingthe wand 103 operation, including coordinating the scanning and inreviewing and processing the scanning and generation of the 3D modelincluding surface and internal features. As shown in FIG. 1B the one ormore processors 113 may include or may be coupled with a memory 115 forstoring scanned data (surface data, internal feature data, etc.).Communications circuitry 117, including wireless or wired communicationscircuitry may also be included for communicating with components of thesystem (including the wand) or external components, including externalprocessors. For example the system may be configured to send and receivescans or 3D models. One or more additional outputs 119 may also beincluded for outputting or presenting information, including displayscreens, printers, etc. As mentioned, inputs 121 (buttons, touchscreens,etc.) may be included and the apparatus may allow or request user inputfor controlling scanning and other operations.

FIG. 2 illustrates one example of a scanner that may be used. Thescanner shown may be used as part of a system such as the system shownin FIGS. 1A-1B. For example, FIG. 2 shows a schematic of intraoralscanner configured to do both surface scanning (e.g., visible light,non-penetrative) and penetrative scanning using a near infra-red (NIR)wavelength (at 850 nm in this example). In FIG. 2, the scanner includesa near-IR illumination light 289 and a first polarizer 281 and a secondpolarizer 283 in front of the image sensor 285 to block near-IR lightreflected off the surface of the tooth 290 (P-polarization light) whilestill collecting near-IR light scattered from internal toothstructures/regions (S-polarization light). The NIR light illuminates thetooth in P-polarization, and specular light reflected from the surfaceof the tooth, e.g., the enamel, is reflected with specular reflectionhence its P-polarization state is conserved. Near-IR light penetratingthe internal tooth features, such as the dentin, is scattered resultingin random polarization (S and P). The wavelength selective quarterwaveplate 293 does not modify the polarization of the near-IR light(e.g., it leaves the polarization state of the near-IR light beingdelivered unchanged) but changes the polarization of the returning scanlight from P to S such that only surface reflection are captured in thescan wavelength. The returning near-IR light, having a mixture of S andP polarizations, is first filtered through the polarization beamsplitter (PBS) 294 and polarizing filter 283 such that only theS-polarization is transmitted to the image sensor. Thus only the near-IRS-polarization light, coming from the tooth internal structures, iscaptured by the image sensor while specular light, having the originalp-polarization, is blocked. Other intraoral scanner configurations withor without polarization filters such as those shown in FIG. 2 may beused as part of the probe.

In FIG. 2, the surface scan may be performed by illuminating the surface(using the scanner illumination unit 297), illuminating inp-polarization, and the polarization may be reversed by thewavelength-selective quarter waveplate 293 (transmitting S-polarizationlight to the image sensor).

A variety of penetrative scanning techniques (penetration imaging) maybe used or incorporated into the apparatuses described herein forperforming scans that to detect internal structures using a penetrativewavelength or a spectral range of penetrative wavelengths, including,but not limited, to trans-illumination and small-angle penetrationimaging, both of which detect the passage of penetrative wavelengths oflight from or through the tissue (e.g., from or through a tooth orteeth). Thus, these apparatuses and techniques may be used to scanintraoral components such as a tooth or one or more teeth, gingiva,palate, etc. and used to generate a model of the scanned area. Thesemodels may be generated in real time or after scanning. These models maybe referred to as 3D volumetric models of the teeth, but may includeother regions of the jaw, including the palate, gingiva and teeth.Although the methods and apparatuses described herein typically relateto 3D volumetric models, the techniques and methods described herein mayalso be used in some instance with 3D surface models. The surface modelinformation is typically part of the 3D volumetric model.

FIG. 3 illustrates an example of a data flow for scanning teeth with anintraoral scanner to build a 3D model including internal structures. InFIG. 3, the exemplary method shown may include three parts. First, theteeth may be scanned with an intraoral scanner 1701 (or any otherscanner) configured to provide penetrative scans into the teeth using anoptical (e.g., IR, near IR, etc.) wavelength or range of wavelengths.Any of these scanners may also concurrently scan to determine a surfacefeatures (e.g., via one or more non-penetrative wavelengths), color,etc., as described above. During scanning, a plurality of penetrativescans 1703, 1703′ may be taken, and the position of the sensor (e.g.,camera) 1705, 1705′ (e.g., x,y,z position and/or pitch, roll, yawangles) may be determined and/or recorded for each penetrative image. Insome variations, the surface of the teeth may also and concurrently beimaged, and a 3D surface model of the teeth 1707 determined, asdescribed above. In this example, the patient's teeth may be scanned,for example, with an intraoral 3D scanner 1702 that is capable ofimaging the inner teeth structure using, for example, near infra-redimaging. The location and orientation of the camera may be determined,in part, from the 3D scanning data and/or the 3D teeth surface model1707.

Thereafter, the penetrative images may be segmented 1711. In thisexample, segmentation may be done in one of two ways. On the inner teethstructure images, the images may be segmented using contour finding1713, 1713′. Machine learning methods may be applied to further automatethis process. Alternatively or additionally, near images (where theircamera position is close) may be used to decide on close features, andalso project features from the 3D model back to the images in order tolocate correctly segments like enamel. The method may also includeprojecting pixels from the inner teeth images back to the teeth andcalculating a density map of inner teeth reflection coefficient.Enclosing surfaces of different segments may be found or estimated byusing iso-surfaces or thresholds of the density map and/or by machinelearning methods. In addition, segmenting the images and projecting thesegments back to a model (such as the 3D surface model, e.g., projectingback to the world), may be used to find a segment by the intersection ofthe segment projections and the teeth surface.

The results may be displayed 1717, transmitted and/or stored. Forexample, the results may be displayed by the scanning system during theintraoral scanning procedure. The results may be shown by images withenclosing contours for different segments, a 3D density map, etc. In theexample shown in FIG. 3 a density map 1715, representing the dentinbeneath the enamel on the outer surface, is shown. This image may becolor coded to show different segments. In this example, internalsegments (structures) are shown within the 3D surface model (which isshown transparent); not all teeth have been scanned with penetrativeimages, thus, only some are shown. Alternative views, sections, slices,projections or the like may be provided. In FIG. 3, the example imageincludes artifacts that are present outside of the teeth 1716; these maybe removed or trimmed, based on the surface model 1718.

A segment may mark each pixel on the image. Internal structures, such asdentin, enamel, cracks, lesions, etc. may be automatically determined bysegmentation, and may be identified manually or automatically (e.g.,based on machine learning of the 3D structure, etc.). Segments may bedisplayed separately or together (e.g., in different colors, densities,etc.) with or without the surface model (e.g., the 3D surface model).

Thus, in FIG. 3, the patient is initially scanned with a 3D scannercapable of both surface scanning and penetrative scanning (e.g., near IRimaging), and the orientation and/or position of the camera is known(based on the position and/or orientation of the wand and/or the surfacescans). This position and orientation may be relative to the toothsurface. The method and apparatus may therefore have an estimate of thecamera position (where it is located, e.g., x,y,z position of thecamera, and its rotational position).

In general, penetrative images (e.g., near IR or IR images) may besegmented automatically.

User Interface and Display of Volumetric Information

The collection and analysis of volumetric data from the intraoral cavitymay identify features and information from teeth that were previouslydifficult or impossible to identify from non-volumetric scanning.However, it may be difficult or non-intuitive for a dental practitioner(and/or patient) to analyze three-dimensional volumetric information.Described herein are methods and apparatuses for viewing andinterpreting 3D, volumetric data of a patient's oral cavity.

For example, FIGS. 9A-9G illustrate one example of a method fordisplaying 3D volumetric data. FIG. 9A shows a surface model (which maybe a surface model portion of a volumetric model) from a top view of anupper arch, in which external features are visible (e.g., surfacefeatures). This view is similar to the surface scan view which may be incolor (e.g., taken by visible light). Internal structures, which arepresent within the model beneath the external surface of the scan, arenot readily visible in FIG. 9A. FIG. 9B, the internal structures areshown based on their relative transparency to near-IR light. In FIG. 9B,the enamel is more transparent (and is shown as more transparent) thanthe dentin, which is shown as less transparent. FIGS. 9B-9F illustrate atransition between the surface view of FIG. 9A and the penetrative,internal 3D view of FIG. 9B for a sub region (circled region “C”) shown.For example, a user display may be provided in which the relativesurface vs. internal views may be altered to provide a sense of internalstructures within the dental arch relative to surface structures. Forexample, an animated view cycling through image such as FIGS. 9C-9G maybe provided. Alternatively, the user may slide a slider 903 togglingbetween the surface and internal views. The transition between these twoviews (which may be made from any angle, may help the user and/orpatient to see beneath the surface of the teeth, to visually assess therich internal data. The 3D volumetric model may be manipulated to showany view, including cross-sectional views, showing internal structuresand/or surface features. In FIG. 9A-9G the top view is shown. FIGS.9C-9G illustrate progressively more transparent views or a region (“C”)of the 3D volumetric model of FIG. 9A in which progressively largepercentages (from 0% to 100%) of the internal view of FIG. 9B is shownfor region C, while progressively less of the surface view (from 100% to0%) is shown.

FIGS. 11A-11C illustrate another example, showing a hybrid image that(like FIG. 9E combines and mixes both surface image scanning (e.g., avisible light scan, as shown in FIG. 11A) with a volumetric model takenusing a penetrative (e.g., near-IR) wavelength, as shown in FIG. 11B.Features that are present in the tooth enamel and dentin are visible inthe volumetric reconstruction (image shown in FIG. 11B) that are notapparent in the image (which may also be a reconstruction) of just thesurface shown in FIG. 11A. For example, in FIG. 11B, a carries region1103 is apparent, which is not visible in FIG. 11A. Similarly a bubbledregion of the enamel 1105 is visible in FIG. 11B but is not visible inFIG. 11A. FIG. 11C shows a hybrid image of the 3D volumetric model andsurface model (surface image), in which both of these structures, thecarries and the bubbled region, are visible.

In general, described herein are methods and apparatuses for simplifyingand displaying volumetric data from a patient's oral cavity (e.g.,teeth, gingiva, palate, etc.) in a manner that may be easily understoodby a user (e.g., a dental practitioner) and/or a patient. Also describedherein are methods of displaying volumetric data taken from a patient'soral cavity in a manner that may be familiar for a user and/or patientto understand. In a first example, the data may be presented as one or aseries of x-ray type images, similar from what would be produced bydental x-rays. FIG. 4 illustrates one method of generating x-ray (orpseudo x-ray) images from a volumetric data set taken as describedabove, e.g., using penetrative light (e.g., near-IR) wavelength(s).

As shown in FIG. 4, a method of displaying 3D volumetric images of apatient's oral cavity may include receiving the 3D volumetric data 401,e.g., from a scan as described above, either directly or from a storeddigital scan, etc. In some variations, individual teeth or groups ofteeth may be identified from the volumetric data 403. The teeth may beidentified automatically (e.g., by a segmenting the volume, by machinelearning, etc.) or manually. Alternatively, the entire volume may beused. Pseudo-x-ray images may then be generated from the volume orsub-sets of the volume corresponding to individual teeth 405. Forexample, an image of the volume may be taken from the ‘front’ of thetooth or teeth, in which the transparency of the enamel (and/orenamel-like restorations), dentin and other features are kept from thevolumetric data. This volumetric data may be based on the absorptioncoefficients of the material within the oral cavity for the penetratingwavelength of light used. Thus, a projection through the volumetric datamay be generated for a fixed direction from the volumetric data to getan image similar to an X-ray, but, in some variations, inverted andshowing the density of the dentin (highly absorbing) as “darker” thanthe density of the enamel (less absorbing and therefore moretransparent); caries may also show up as more absorbing (darker)regions. The image may therefore be inverted to resemble an x-ray imagein which more dense regions are lighter (e.g., brighter). These pseudox-ray images may be generated from the same positions as standard dentalx-rays and presented to the user. For example, a panel of pseudo x-rayimages may be generated from the volumetric model for each of thepatient's teeth. Although the penetration of the wavelength of the light(e.g., near IR light) may not be as deep as with traditional x-rays,images generated in this manner may provide a comparable proxy for anx-ray, particularly in the crown and mid-tooth regions above thegingiva.

Other simplified or modified displays may be provided to the user, orcustomized for display by the user to the patient. For example, in somevariations images of the teeth may be generated from the volumetric datain which the image is simplified by pseudo coloring the volumetric datato highlight certain regions. For example, regions that have beenpreviously marked or flagged (as will be described in greater detail,below) may be colored in red, while the enamel may be shown as a morenatural white or slightly off-white color. In some variations,enamel-like materials (e.g., from fillings, etc.) may be representedseparately and/or marked by a color, pattern, etc.

In some variations, the methods and/or apparatuses may display the teethin sections through the dental arch. Similarly the individual teeth orgroups of teeth may be shown separately and/or labeled (e.g., bystandard naming/numbering convention). This may be shown in addition orinstead of other displays. In some variations, the teeth and/or internalstructures may be pseudo-colored or projecting on to a color image maybe used.

For example, FIGS. 5A and 5B illustrate virtual sections taken through avolumetric model of a patient's teeth generated from an intraoral scanthat included near-IR information, and described above. In FIG. 5A, thecross-sectional view may be generated automatically or manually, e.g.,by the user, to display regions of interest within teeth, includingenamel. The cross-section may show both density sectioning and/orsurface sectioning. These images may be pseudo-colored to show differentregions, including outer surfaces, enamel, dentin, etc. Internalstructures, e.g., within the enamel and/or dentin, may reflect theeffect of the near-IR light within the teeth, such as the absorptionand/or reflection of light at one or more near-IR/visible wavelengthswithin the teeth. In FIG. 5B, the section is pseudo-colored with a heatmap to show internal features, and a key may be provided, as shown. Inany of these variations, 2D projections of the teeth may be generatedfrom the volumetric information, showing one or more features on thetooth and/or teeth. As will be described in greater detail below,additional features, including lesions (e.g., caries/cavities, cracks,wearing, plaque build-up, etc.) may be displayed as well, and may beindicated by color, texture, etc. While illustrated as sections of the3D volumetric model, other embodiments may display the 2D section byitself to provide a cross-sectional view of the tooth/teeth similar to aview provided by 2D x-ray images.

Any of the methods and apparatuses for performing them described hereinmay include displaying one or more (or a continuous) sections through a3D model of the patient's dental arch, and preferably a 3D volumetricmodel. For example, a method of displaying images from athree-dimensional (3D) volumetric model of a patient's dental arch mayinclude: collecting the 3D volumetric model of the patient's dentalarch, wherein the 3D volumetric model includes near-infrared (near-IR)transparency values for internal structures within the dental arch;generating a two-dimensional (2D) view into the 3D volumetric modelincluding the patient's dental arch including the near-IR transparencyof the internal structures; and displaying the 2D view. In any of thesemethods, the method may optionally (but not necessarily) includescanning the patient's dental arch with an intraoral scanner.

Generating the 2D view may comprises sectioning the 3D volumetric modelin a plane through the 3D volumetric model. The user may select theplane's location and/or orientation, and my do this in a continuousmanner. For example, any of these methods may include selecting, by auser, a section though the 3D volumetric model to display, whereinselecting comprises continuously selecting sections through the 3Dvolumetric model as the user scans through the 3D model and continuouslydisplaying the 2D views corresponding to each section. Generating the 2Dview may comprises selecting, by a user, an orientation of the 2D view.

In any of these methods, the surface may be included. For example, asdescribed and illustrated above, a method of displaying images from athree-dimensional (3D) volumetric model of a patient's dental arch mayinclude: collecting the 3D volumetric model of the patient's dentalarch, wherein the 3D volumetric model includes surface values andnear-infrared (near-IR) transparency values for internal structureswithin the dental arch; generating a two-dimensional (2D) view into the3D volumetric model including the patient's dental arch including bothsurface values and the near-IR transparency of the internal structures;and displaying the 2D view. The surface values may comprise surfacecolor values. The surface relative to the internal (volumetric)structures may be weighted. For example, generating the two-dimensional(2D) view through the 3D volumetric may also include including in the 2Dview a weighted portion of the surface values and a weighted portion ofthe near-IR transparency of the internal structures. The weightedportion of the surface values may include a percentage of the full valueof the surface values, and the weighted portion of the near-IRtransparency of the internal structures comprises a percentage of thefull value of the near-IR transparency of the internal structures,wherein the percentage of the full value of the surface values and thepercentage of the full value of the near-IR transparency of the internalstructures adds up to 100%. For example, the user may adjust theweighted portion of one or more of the surface values and the near-IRtransparency of the internal structures.

For example, a method of displaying images from a three-dimensional (3D)volumetric model of a patient's dental arch may include: collecting the3D volumetric model of the patient's dental arch, wherein the 3Dvolumetric model includes surface color values and near-infrared(near-IR) transparency values for internal structures within the dentalarch; selecting, by a user, an orientation of a view of the 3Dvolumetric model to display; generating a two-dimensional (2D) view intothe 3D volumetric using the selected orientation, including thepatient's dental arch including a weighted portion of the surface colorvalues and a weighted portion of the near-IR transparency of theinternal structures; and displaying the 2D view.

In addition to displaying qualitative images of the teeth, the methodsand apparatuses described herein may quantify, and may providequantitative information about internal and/or external features. Forexample, volumetric measurements of one or more lesions may be provided(selectably or automatically) including dimensions (peak or mean length,depth, width, etc.), volume, etc. This may be performed by manually orautomatically segmenting the volumetric model to define the regions ofinterest, including either or both tooth features (enamel, dentin, etc.)and/or irregularities (e.g., caries, cracks, etc.). Any appropriatesegmentation technique may be used, such as but not limited to: meshsegmentation (mesh decomposition), polyhedral segmentation,skeletonization, etc. Once the volume has been segmented, these regionsmay be separately or collectively displayed and/or measured. As will bedescribed below, they may also be marked/flagged and used for furtheranalysis, display and modification of the scanning methods and systems.

In some variations of the user interfaces described herein, a summaryreport may be generated or created and displayed for the user and/orpatient from the volumetric data. For example, summary data may beprojected onto a model of the patient's teeth. The model may also besimplified, so that the enamel is opaque, but marked or selectedinternal features (including automatically selected internal features)are shown in red or some other contrasting color (and/or flashing,blinking, etc.) within the tooth. For example, caries may be shown inthis manner. The summary report may be automatically entered into apatient chart.

Any of the images, including the volumetric images, may be animated. Forexample, virtual sections through the patient's teeth, showing ascanning or traveling cross-section through the patient's dentition maybe shown, in some cases with a 3D model showing one or more cutting axesthrough the volume. The user interface may allow the user to section inone or more planes, showing both external and internal features based onthe volumetric scan.

In general, the apparatuses described herein may generate separate viewfor the user (e.g., physician, dentist, orthodontist, etc.) than thepatient. The user may be provided with a ‘clinical view’ that mayinclude information not present on a separate ‘patient view.’ Theclinical view may be more technical, and may in some cases be closer tothe raw images from the volumetric data. The user may select whichlayers of information to include in the patient view, which may then bepresented to the user during or after the scanning or review of thedental scanning. Patient educational materials may be appended to thepatient view.

For example, in some variations, the user display of volumetric data mayinclude an overlay of the volumetric data in which pseudo coloring ofthe 3D components within the volumetric data is shown. As will bediscussed in more detail below, in any of these displays/images markedor highlight regions may be shown to call attention to potential problemregions (e.g., caries, thin enamel, cracks, etc.). Two-dimensional (2D)color data and 3D near-IR data (e.g., surface and volumetric regions)may be shown, including transitions between the two.

In general, the volumetric information may be annotated (e.g., marked,labeled, etc.) either automatically, manually, or semi-automatically,and this annotation may be displayed. Furthermore, annotations may beused both to annotate future additional scans, and to modify how futurescans of the same patient are taken and displayed. An annotation may be,for example, a marker or flag on a region of interest. Regions ofinterest may correspond to specific regions in which one or morefeatures (cracks, caries, thinning of enamel, buildup of plaque orcalculus, etc.) have been observed. Alternatively or additionally,regions of interest may be regions in which there is a change over time,e.g., from one scan to another scan.

As mentioned above any of these methods may include placing one or moremarkers on the volumetric model of the patient's teeth. Markers (e.g.,flags, pins, etc.) may be manually placed by the user, or may beautomatically placed by the apparatus, or may be semi-automaticallyplaced (e.g., suggested by the system, configured by the user, etc.).This is described in greater detail below.

Markers may be used to focus attention and/or processing by the systemon one or more specific regions of the volumetric model for display,and/or for later follow-up (e.g., in future scans). Markers may modifythe manner in which the later scans are taken, e.g., taking future scansof marked regions with greater detail (e.g., higher resolution,different wavelengths, greater scanning frequency or reputations, etc.).Marked regions may be displayed over time to show changes in the markedregions.

For example, a user can mark a digital representation of the patient'steeth (or the patient's actual teeth) with a marker (e.g., a pin, flag,etc.) which can be annotated (e.g., can have notes associated with it).This marker may then be used to track over time between different scans.Later scans can be marked in the corresponding location, the later scancan be modified based on the marked regions. These marked regions may bescanned in greater detail, and analytics may be automatically performedand/or displayed, measuring and/or indicating a change compared to oneor more earlier scans. Thus any of the systems described herein maytrack one or more marked regions from previous scans and give feedbackduring and/or after a new scan, providing additional detail. This can bedone for both surface and/or volumetric information, particularly on theproperties of the enamel, and/or by comparison to the enamel, the outersurface of the teeth/tooth, and/or the dentin.

For example, one or more annotation markers from an earlier scan maymodify a subsequent scanning of the same patient. Before scanning, theuser may enter an identifier of the patient being scanned (alternativelythe system may automatically identify the patient based on a database ofearlier scans). The system may automatically annotate the new scan basedon the prior scan annotations.

In some variations the later scans may be automatically annotated by thesystem by identifying differences between the prior scan(s) and thecurrent scan. For example, regions showing a change above a thresholdcompared to the earlier scan may be flagged and presented to the user.The annotation may be done without user oversight (fully automatic) ormay be done with some user oversight, for example, by flagging it andindicating to the user why it was flagged, then allowing the user tokeep, modify or reject the marking. Reasons for automatically markingthe teeth may include a change in the enamel thickness, a change in thesurface smoothness, a change in the relative ratio of enamel vs. dentinin a tooth, a change in the position of the tooth (e.g., occlusion),etc. It may also include a change in structures external to the naturaltooth, such as increase or decrease in plaque or calculus buildup, orchanges to the gingival structures surrounding the tooth, Thus, if thesystem detects one or more of these conditions, it may automaticallyflag the relevant region in the volumetric model.

Later scans may be dynamically modified by the flags from earlier scansor by a detection of a change in a region (even unmarked regions)compared to earlier scans. For example, the scanning parameters may bemodified to scan at higher resolution (e.g., changing the scan dwelltime, requiring the user to scan this region multiple times, etc.),changing the wavelength used for the scanning, etc.

For example, FIG. 6 illustrates a method of automatically selecting aregion for marking and/or using the selected regions. A first volumetricmodel of the patient's teeth is generated from a scan of the patient'steeth 601. The volumetric model may be generated using any appropriatemethod, including those described above and discussed in U.S. patentapplication Ser. No. 15/662,234, filed Jul. 27, 2017, titled “INTRAORALSCANNER WITH DENTAL DIAGNOSTICS CAPABILITIES”, incorporated by referencein its entirety. The first volumetric model may be stored (digitallystored) as part of the patient's dental record. The first volumetricmodel may concurrently or subsequently (immediately or some timethereafter) be analyzed (e.g., by an apparatus, including an apparatushaving a processor that is configured to operate as described herein) toidentify any regions that should be flagged 603. This analysis maytherefore be performed automatically, and may examine one or moreproperties of the patient's teeth from the scan. Automatic (orsemi-automatic, etc., automatic but with manual assistance toverify/confirm) may be performed by a microprocessor, including systemsthat have been trained (e.g., by machine learning) to identify regionsof irregularities on the outside and/or internal volume of the teeth.For example, the apparatus may examine the digital model to identifypossible defects in the patient's teeth such as (but not limited to):cracks, caries, voids, changes in bite relationship, malocclusions, etc.This may include identifying regions in which there is an optical (e.g.,near-IR) contrast near the surface of the tooth indicating a possiblecrack, caries, occlusion, etc. 605. Regions that are less transparent(e.g., more absorptive) in the near-IR wavelengths than the rest of theenamel, that are closer to the surface (generally or within specificnear-IR wavelengths), may correspond to defects. Alternatively oradditionally, surface properties of the teeth may be examined andflagged if they are outside of a threshold 607. For example, regions ofthe surface of the teeth in which the tooth surface is rough (e.g., hasa smoothness that is below a set threshold, where smoothness may bedetermined from the outer surface of the enamel) may be flagged. Othersurface properties may also be analyzed and used to determine if theregion should be marked or presented to the user to confirm marking,including discoloration (based on a color or white-light/surface scan),gingival position (relative to the outer surface of the tooth), etc. Thedistribution and size of the patient's enamel may also be examined 609.The enamel thickness may be determined from the optical properties(e.g., comparing absorption/reflectance properties). Regions of putativeenamel that are below a threshold thickness, or having a ratio ofthickness to tooth dimension (e.g., diameter, width, etc.) below athreshold may be marked or presented to the user to confirm marking.

In some variations, during and/or after automatically analyzing thevolumetric model, the use may also manually flag one or more regions ofthe volumetric model of the patient's teeth 615. If the automaticanalysis of the volumetric model automatically flags the identifiedvolumetric model the user's manually added regions may be added. In somevariations, the user may be prompted to flag the regions identified andsuggested by the automatic analysis. These regions may be marked and anindication of the reason(s) for their being identified may be provided(e.g., irregular enamel, potential crack, potential caries, potentialthinning of the enamel, etc.). In general, the inner boundary (theboundary within the volume of the tooth, for example) may be defined inany of the methods and apparatuses described herein. For example, invariations in which a region of the enamel is thinning, the methods andapparatuses described herein may be used to the entire layer (e.g.,layer of enamel, region of the inner structure of the tooth) may beidentified and used for qualitative and/or quantitative information.

In some variations the method flagged regions may then be displayed on adigital model of the patient's teeth 617. The display may emphasize theflagged regions, e.g., by providing a color, animation (e.g., flashing),icon (e.g., arrow, flag, etc.), or the like, including combinations ofthese. The display may also show enlarged views of any of these. Theuser may modify the display, e.g., rotating, sectioning, enlarging,etc., the flagged region. Alternatively or additionally, the flaggedregions may be enlarged on the display by default. An index or key ofthe flagged regions may be provided, and may be displayed and/or storedwith the digital volumetric model of the patient's teeth.

In some variations, as shown to the right of FIG. 6, the method mayinclude using the flagged regions to modify the future scans, asmentioned above. For example, the method may include scanning(“rescanning”) using the flagged regions, after some interim time periodof between a first time (e.g., about one day, one week, one month, 2months, 3 months, 4 months, 5 months, 6 months, 7 months, 8 months, 9months, 10 months, 11 months, 1 year, 1.2 years, 1.5 years, 2 years,etc.) and a second time (e.g., about one month, 2 months, 3 months, 4months, 5 months, 6 months, 7 months, 8 months, 9 months, 10 months, 11months, 1 year, 1.5 years, 2 years, 3 years, 4 years, 5 years, 6 years,7 years, 8 years, 9 years, 10 years, 11 years, 12 years, 13 years, 14years, 15 years or more, etc.), or longer than the second time period.The flagged regions may be used to modify the scan by increasing theresolution of the scanned regions during the scan, e.g., by increasingthe scan rate, increasing the dwell time in this region, scanning theregion in additional wavelengths, scanning this region multiple times,etc. The scanning apparatus may inform the user to adjust the scanning(e.g., moving the wand of an intraoral scanner more slowly in theseregions, moving the scanner back over these regions multiple times,etc.) and/or it may automatically adjust the scanning parameters duringoperation. The scanning apparatus may therefore receive the key or indexof flagged regions and/or a marked (flagged) version of the patient'searlier intraoral scan(s). Prior to scanning the scanning apparatus, theuser may indicate the identity of the patient being scanned, and thismay be used to look up the earlier scan(s). Alternatively oradditionally, the apparatus may identify the patient based on thecurrent scan to identify (or confirm the identity) of the patient, toverify or recall the earlier annotated (flagged) scan. Alternatively,the second or subsequent scans may be taken without using the earlierflagged regions.

Following the subsequent (e.g., second, later or follow-up scan orscans), the method or apparatus configured to perform the method maythen compare the flagged regions from the subsequent scan with thecorresponding regions from the earlier scan(s) 621. In addition, thevolumetric model from the subsequent scan may automatically analyzed toidentify any regions of the new (subsequent) scan that can/should bemarked/flagged (e.g., repeating the earlier automatic or semi-automaticanalysis steps 603-615) 621. Newly identified regions from thesubsequent scan may be compared to the previously un-flaggedcorresponding regions in the earlier volumetric model(s).

The flagged regions may be analyzed over time 623. Specific sub-regionsfrom the volumetric model including the flagged regions may be generatedfor display and analysis. The results may be output 625. For example,these regions may be displayed to the user along with descriptive,analytic information about the scanned region. These regions may also bemarked to shows changes over time. The data may be displayed in ananimation, for example, showing changes over time. In some variations,the images may be displayed as a time-lapse image (video, loop, etc.),showing changes. Time-lapse images may show the change in the internaland/or external structure over time. Sections (pseudo-sections generatedfrom the volumetric model(s)) may be used to show changes. Color,texture, patterns, and any other highlighting visualization techniquemay be used. Alternatively or additionally to the display of the flaggedregions, these regions (and any accompanying analysis) may be output inother appropriate ways, including digitally outputting (e.g., thepatient's dental record), printing a description of the flaggedregion(s), etc.

Any of the methods of tracking a region of a patient's dental archdescribed herein may include tracking over time and/or across differentimaging modalities (e.g., records) as described in greater detail below.Further, any of these methods may be automated and/or may includeautomated agents, for example, for identifying one or more regions ofinterest (e.g., features, defects, including actionable dentalfeatures), including for scoring them and/or automating identification,scoring and/or display of such regions. Any of these methods may alsoinclude any of the display methods or agents (e.g., for displayingsections, displaying internal structures, for displaying virtual x-rays,for displaying across imaging modalities, etc.).

For example, a method of tracking a region of a patient's dental archover time may include: collecting a first three-dimensional (3D)volumetric model of the patient's dental arch, wherein the 3D volumetricmodel includes surface values and near-infrared (near-IR) transparencyvalues for internal structures within the dental arch; identifying aregion of the 3D volumetric model; flagging the identified region;collecting a second 3D volumetric model of the patient's dental arch;and displaying one or more images marking, on the one or more images, adifference between the first 3D volumetric model and the second 3Dvolumetric model at the flagged region.

Any of these methods may also include tracking and/or comparing acrossdifferent records (e.g., different imaging modalities), so thatidentifying comprises identifying a region of the patient's dental archfrom a first record of a plurality of records, wherein each recordcomprises a plurality of images of a patient's dental arch each takenusing an imaging modality, further wherein each record of the pluralityof records is taken at a different imaging modality, further whereinflagging comprises flagging the identified region in a correspondingregion of the 3D volumetric model of the patient's dental arch. Themethod and apparatuses for performing them may also include correlatingthe flagged region with each of records of the plurality of records bycorrelating the 3D volumetric model of the patient's dental arch witheach of the records of the plurality of records. In some variations thecorrelation may be used to weight or grade the identified region todetermine if it corresponds to a region of interest (e.g., a feature, adefect, including actionable dental features, etc.). For example, theregion of the patient's dental arch may comprises a dental featurecomprises one or more of: cracks, gum recess, tartar, enamel thickness,pits, caries, pits, fissures, evidence of grinding, and interproximalvoids. Identifying the region may comprise comparing a near-IRtransparency value of a region within the 3D model to a threshold value.

Where surface values are used, the surface values may comprise surfacecolor values. These methods may be used with stored data and/or withdata collected in real time (e.g., thus the method may optionally butnot necessarily collect a three-dimensional (3D) volumetric model byscanning the patient's dental arch to generate the 3D volumetric model.

Identifying the region may comprise comprises automatically identifyingusing a processor. For example, automatically identifying may compriseidentifying a surface color value outside of a threshold range.Automatically identifying may comprise segmenting the 3D volumetricmodel to identify enamel regions and identifying regions having enamelthicknesses below a threshold value.

Flagging the identified region may comprise automatically flagging theidentified regions or manually confirming the identified region forflagging.

In any of these method in which regions are flagged, the method mayinclude re-scanning the patient's dental arch wherein the flagged regionis scanned at a higher resolution than un-flagged regions.

A method of tracking a region of a patient's dental arch over time mayinclude: collecting a first three-dimensional (3D) volumetric model ofthe patient's dental arch taken at a first time, wherein the 3Dvolumetric model includes surface color values and near-infrared(near-IR) transparency values for internal structures within the dentalarch; identifying, using an automatic process, a region within the 3Dvolumetric model to be flagged from a first record of a plurality ofrecords, wherein each record comprises a plurality of images of apatient's dental arch each taken using an imaging modality, furtherwherein each record of the plurality of records is taken at a differentimaging modality; flagging the identified regions; correlating theflagged region with each of the records of the plurality of records bycorrelating the 3D volumetric model of the patient's dental arch witheach of the records of the plurality of records; collecting a second 3Dvolumetric model of the patient's dental arch taken at a separate time;and displaying a difference between the first 3D volumetric model andthe second 3D volumetric model at the flagged region.

Similarly, as summarized and described above, a method of tracking adental feature across different imaging modalities, the methodcomprising: collecting a first three-dimensional (3D) volumetric modelof the patient's dental arch, wherein the 3D volumetric model of thepatient's dental arch includes surface values and internal structureswithin the dental arch; identifying a region of the patient's dentalarch from a first record of a plurality of records, wherein each recordcomprises a plurality of images of a patient's dental arch each takenusing an imaging modality, further wherein each record of the pluralityof records is taken at a different imaging modality; flagging theidentified region in a corresponding region of the 3D volumetric modelof the patient's dental arch; correlating the flagged region with eachof the records of the plurality of records by correlating the 3Dvolumetric model of the patient's dental arch with each of the recordsof the plurality of records; and saving, displaying and/or transmittingimages including the region of the patient's dental arch. Any of thesemethods may include tracking over time as well, e.g., by comparing thesame region(s) to 3D volumetric models at different times.

FIGS. 10A and 10B illustrate a user interface showing marking of aregion of interest from a 3D volumetric scan of a patient's oral cavity.In FIG. 10A, the user interface includes an image of the internalfeatures 1001 (e.g., based on the near-IR absorption of the teeth),similar to FIG. 9B, discussed above. This view may be manipulated byuser controls 1015, including sectioning tools, rotation, moving tools,etc. In FIG. 10A, two upper windows show a surface view 1003 and avolumetric (internal) view 1005 corresponding to the same region. Thisregion may be selected. FIG. 10B shows the same features of FIG. 10A,but with a region marked or flagged 1011. As discussed above, theidentification of the region to be flagged may be automatic or manual,or semi-automatic (e.g., confirmed by the user), and may be chosen toselect a region for later monitoring. In FIG. 10B, the region maycorrespond, for example, to possible caries.

Monitoring of one or more internal regions of the teeth over time usingthe volumetric models of the patient's teeth taken with the devicesdescribed herein may be particularly helpful for predicting dentalproblems, including caries, cracks, tooth loss, gum recession, and thelike. In particular, these methods and apparatuses may help the user(e.g., dentist, dental technician, orthodontist, etc.) inform andeducate a patient so that the patient may take recommended treatmentsprior to developing more serious problems. There is a need for effectiveways to show changes in teeth over time and to provide patients withinformation necessary to act early to prevent more complicated andpotentially painful problems from developing. Many patients areotherwise reticent to undergo preventive procedures, particularly whenthere is not currently any associated pain or discomfort. For example,pre-cavitated caries are difficult to identify with current imagingtechniques, and it may be particularly difficult to convince a patientto treat even when identified, since they typically present withoutpain. However, early stage treatment may be critical to avoiding morecomplicated and dangerous procedures later.

Dental caries are one type of problem that may be identified with themethods and apparatuses described herein. As shown and discussed above,caries may be identified from 3D volumetric models (such as thosedescribed herein) that penetrate into teeth using light (e.g., near-IR),one type of non-ionizing radiation. In the 3D volumetric modelsgenerated as described herein, e.g., using a near-IR light, typically incombination with a surface scanning (e.g., white light), the absorptioncoefficients of the internal regions of the teeth may indicatedistinctions between dentin and enamel, and may reveal internalstructures and flaws, including cracks, caries and the like. Forexample, regions of enamel that are less transparent than expected inthe near-IR wavelengths may (e.g., having different IR opticalproperties), and particularly those that appear to extend to the surfaceof the tooth in the volumetric model may be identified manually orautomatically as cavities or caries. Other irregularities in the enameland/or dentin (e.g., based on the internal features of the teeth fromthe volumetric model) may be identified and may be characteristic of aproblem in the teeth. Thus, the techniques described herein may be usedfor prognosis of dental issues such as caries.

As mentioned, any of the apparatuses and methods described herein mayinclude improved methods for displaying of internal tooth features usingthe one or more volumetric models of a patient's teeth. For example, themethods and apparatuses described herein may be used to generate anestimate of enamel thickness for one or more of the patient's teeth.These estimates may be visually displayed, showing the outer surface ofthe teeth or a particular tooth, and may also show internal structures,including in sectional views or 3D internal views showing, e.g., theenamel, including the thickness of the enamel. This information may beused clinically to determine the need for, to help design and to helpapply dental prosthetics, including veneers, crowns, and the like. Anyof the methods and apparatuses described herein may be used, forexample, to help prepare design a dental implant for a particular toothor teeth.

Plaque and Calculus Detection and Visualization

The method and apparatuses described herein may also be used for thedetection and visualization (including quantification) of plaque andcalculus on the patient's teeth. Any of the intraoral scanners describedherein may be used to detect plaque or calculus on the patient's teethby using florescence imaging in addition to other imaging/scanningmodalities including penetrative (e.g., near-IR) imaging. For example,and intraoral scanner may be cycled between different imaging modalitiessuch as between white light and near-IR, including additional modalitiessuch as florescence (e.g., laser florescence, etc.).

The use of fluorescence capabilities (and/or using current ones) by theintraoral scanner may allow detection of plaque and calculus on thesurface of the teeth. In combination with 3D modeling using the datafrom the intraoral scanner, the plaque/calculus conditions can bemodeled and visualized on the 3D model of the teeth, including the 3Dvolumetric modeling of the teeth. Plaque and/or calculus may be detectedand may be displayed and highlighted as described above, and may be usedbefore, during or after treatment. For example, a dental technician(e.g., dental hygienist) may use an intraoral scanner to detect andmonitor the condition of the patient and the cleaning treatment. Data onplaque and calculus may also be used by any of the apparatuses describedherein to determine and provide predictive models that may indicateplaque and calculus (e.g., tartar) generation rate and/or locations.

In some variations, plaque and calculus may be identified at least inpart using florescence information. It has been observed that plaque mayfluoresce under blue light (e.g., around about 405 nm). Any of theintraoral scanners described herein may include fluorescence informationfrom which information about plaque and calculus may be used, andincorporated into a 3D model of the patient's teeth. For example, plaqueand/or calculus may be visually displayed as a color and/or texture onthe 3D model of the patient's teeth.

For example, a fluorescence signal can be obtained from an intraoralscanner using a dichroic filter having a large aperture amplification offluorescence signal. This amplification may emphasize the fluorescence,thus enabling the detection, visualization and segmentation of plaqueand calculus regions using RGB illumination, sensor and image.Alternatively or additionally, the apparatus may include a florescencesource (e.g., an LED emitting at 405 nm) and corresponding filter(s) fordetection of plaque and/or calculus. This may be integrated in theintraoral scanner, or added (e.g., as a sleeve, accessory, etc.) to beused with the scanner.

Alternatively or additionally, in some variations, depending on thewavelength of near-IR light used, the plaque and calculus may have adifferent absorption/reflection than enamel. This may allow the calculusand/or plaque to be differentiated from the enamel in the volumetricmodel. Further, the volumetric model may be used to detect material onthe teeth, including calculus and plaque based on the surface smoothnessand geometry. In variations in which calculus and/or plaque are nottransparent to the near-IR frequencies used, the apparatus maydifferentiate calculus and/or plaque from the enamel using thevolumetric model. Thus, the calculus and/or plaque may be segmented anddifferentiated from enamel.

The use of an intra-oral scanner to detect plaque and/or calculus mayprovide quantitative information and digital modeling. This may allowmonitoring and comparison of plaque/calculus over time based onregistration to 3D model, including real-time registration and/ordisplay.

The acquisition of both fluorescence image and 3D scan on the same timeand same position of the intraoral scanner (e.g., the scanning wand)allows for very accurate registration of the plaque/calculus regions andthe 3D model. The concurrent scanning is described in greater detail,for example, in U.S. patent application Ser. No. 15/662,234, filed Jul.27, 2017, and titled “INTRAORAL SCANNER WITH DENTAL DIAGNOSTICSCAPABILITIES”. The accurate registration between different scanningmodalities, such a white/visible light, penetrative (near-IR) lightand/or florescence, may enable the apparatuses to define the borders ofthe calculus and/or plaque and may permit the apparatus to determine thevolume/thickness in high resolution, allowing for both measuring theprecise current situation and comparison/tracking relative to previousscans.

The methods and apparatuses described herein may take RGB images of theteeth at the same/similar time with taking 3D scans of the teeth. Thesescans may then be used to build the 3D model of the teeth/jaw, which mayinclude the volumetric information (3D volumetric model). For example,RGB images may show emphasized signal of fluorescent surfaces,specifically plaque and calculus regions, due to specific characteristicof color and brightness of such surfaces, as mentioned. For example, theimage of the outer surface (and in some cases the volumetric model) ofthe teeth may show regions having optical properties (florescence,brightness, color, etc.) indicative of calculus and/or plaque. In somevariations, this emphasized signal may result from the spectralillumination that creates no reflection in visible light, but creates asignificant fluorescence signal from plaque and calculus. For example,typical RGB illumination (using a common RGB sensor), may be modified toprovide amplification of the fluorescence signal (e.g., in near-IRregions) on the outer surface of the teeth. This amplification can beachieved by, as a non-limiting example, a large aperture that enables IRsignals to pass, and small aperture that enables the regular RGB(visible) spectrum to pass. This combination may produce color imageswith extra emphasis on fluorescent surfaces. Such fluorescence maymanifest in characteristic colors and brightness of the desired regionsindicating calculus and/or plaque on the teeth.

In any of the method and apparatuses described in which RGB images maybe take that include florescence signals (e.g., at a wavelength in whichplaque or calculus fluoresces), segmentation of the fluorescent regionsmay be performed on the image. For example, using the camera positionsduring acquisition of RGB and 3D scans (e.g., from the intraoralscanner), the fluorescent region may be registered with the 3D model(including the volumetric and/or just surface model) of the patient'steeth. This may result in a definition of relevant plaque and calculusregions on the final 3D model, which may further allow for definition ofthese regions, such as the borders of the calculus on the teeth, as wellas 3D surface & thickness of the plaque.

As already discussed above, regions on the 3D model may be compared withprevious/future scans of the same patient, which may show thedevelopment of calculus over time, and the effect of the calculus on thepatient's teeth. The apparatus may automatically or semi-automaticallymark (e.g., flag) these regions for monitoring. Thus, the size and shapeof calculus for each tooth may be monitored. Alternatively oradditionally, the thickness/depth of calculus may be compared withprevious scans. Any of this information may be provided quantitativelyand/or qualitatively, as discussed above. The thickness/depth ofcalculus may be compared with previous scans of clean teeth (includingone or more earlier scans following cleaning by a dental professional).This may provide an estimate of the thickness of the calculus in laterscans. As mentioned, measurements of the changes in the plaque, andparticularly the calculus over time may be made, and this data may beused to monitor plaque and calculus progression on the patient's teeth,and may provide well as visualization of the development.

In general, the monitoring and visualization of the patient's teethusing the methods and apparatuses described herein may be used as partof a dental and/or orthodontic treatment planning. As already mentionedabove, monitoring of calculus and plaque may be used to treatmentsincluding teeth cleaning. Scans may be performed prior to cleaning,during cleaning and/or after cleaning to provide guidance to the dentalpractitioner as to what regions to emphasize, focus on, or return to.Other treatments (coatings, caps, etc.) may be proposed based on theprogression of plaque and/or calculus over time. Further, monitoring ofany other feature or region of interest, including, e.g., caries,cracks, etc., as described above, may also provide treatment planninginformation. As discussed above, information about cracks and/or cariesmay be used to suggest treatments including restorations beforepotential issues develop further. In some variations, a digital model(e.g., surface and/or volumetric model) of the teeth may be modifiedusing the volumetric information, and the modified model(s) used todesign an orthodontic appliance or treatment plan. For example, a usermay digitally remove plaque and/or calculus from a volumetric scan takenprior or during a treatment. The modified scan may be used to guidetreatment, including further cleaning of the teeth, as necessary and toform or modify an appliance, so that the appliance (e.g., a dentalaligner) may fit better.

Combination with Dental Tools

The intraoral scanners and volumetric modeling described herein may beused and/or combined with other dental tools (drills, probes, etc.). Thecombined tool may provide numerous advantages.

For example, described herein are drills that may be used in conjunctionor combined with the intraoral scanners, and the use of 3D volumetricmodels. In some variations, a dental drill and an intraoral scanner maybe combined; e.g., incorporating a laser drill or laser-acceleratedwater drill into an intraoral scanner. This combination may allow thedental professional using the tool to directly visualize the tooth asand before it is drilled, providing real-time feedback to the user. Inone example, near-IR light may be applied to the probe head of the drill(e.g., laser drill) to provide imaging into the tooth, which will allowdirect forward-looking imaging prior and/or during drilling. The enameland dentin in the direct path of the drill may be imaged. The densityinformation can be used to inform the clinician when they have reachedthe dentin layer of a tooth or a certain depth inside the dentin, orwhen diseased regions have been removed. For example, the densityinformation can be used to provide haptic feedback to the operator,since tactile feedback is much more limited when using a dental laserversus a traditional headpiece.

The methods and apparatuses including intraoral scanners and volumetricmodeling as described herein may also be integrated into computer-aideddesign/computer-aided manufacturing technology for dentistry, asdescribed in FIG. 7. For example, dental implants, such as crowns (e.g.,ceramic crowns) may be fabricated for an individual patient usingcomputer-aided design and computer-aided manufacturing (CAD/CAM)apparatuses and procedures. For example, traditionally CAD/CAMlaboratory manufacturing (“current workflow” in FIG. 7) may includepre-treatment scanning of the patient's teeth 701 or an impression ofthe patient's teeth, such as a caries-free scan of the jaws. The teethmay then be prepared for the crown 703, and then re-scanned 705 andevaluated 707. Finally, the crown may be made using CAD/CAM. Thesoftware for the CAD/CAM may receive the scanned information from thescanner and may process it for use in forming the design and performingthe manufacture. The use of CAD/CAM software may provide restorationscomparable to conventional restorations in all aspects, including inaesthetics, however the current methodologies may require repeated stepsfor evaluating and preparing the tooth, as shown by FIG. 7, andtypically require the user to perform these steps manually.

As shown in the “new workflow” on the bottom of FIG. 7, this method mayintegrate the 3D volumetric modeling described herein to simplify andimprove CAD/CAM of a patient's teeth. For example, the preparation maybe digitally designed, and this process may be automated (fully orsemi-fully, so that the user may approve and/or modify the process). Forexample, in FIG. 7, the pre-treatment scan 711 may be performed using anintraoral scanner that directly communicates with the CAD/CAM apparatus,or the intraoral scanner may include CAD/CAM capabilities. In thisexample, the tooth preparation 713 may be fully digitally designed basedon the scan performed, and the scanner may guide the preparation of thetooth 715. This may be done in real time with direct feedback and/orguidance from the apparatus, which may integrate the scanner. Thescanner may then be used to evaluate the preparation 717, on in somecases this step may be integrated fully into the guided prep step 715,therefore removing the need for the post-prep evaluation. Finally,CAD/CAM may be used to prepare the crown (or other dental appliance) forthe correctly prepared tooth 719.

Root Canal

The methods and apparatuses for 3D volumetric modeling of the patient'soral cavity (e.g., 3D volumetric modeling of the teeth) may also be usedto modify a root canal procedure. Typically, root canal proceduresrequire numerous x-rays to provide images into the teeth prior to,during, and/or after the procedure. The methods and apparatusesdescribed herein may remove or reduce the requirement for x-rays in thespecific example of root canal procedures. Specifically, as describedherein, an intraoral scanner including penetrative wavelengths (e.g.,near-IR) may be used to examine within a tooth, including within theroot of the tooth during the procedure. This may allow identificationand localization of the canal. For example, a tooth may be prepared forthe root canal by, for example, drilling a hole through the crown intothe tooth. The hole may be drilled with the guidance of the volumetricimaging described herein either during or interposed with the drilling.For example, a tooth (e.g., molar) may have an initial hole drilled intoit to expose the camber within the tooth. An intraoral scanner includingnear-IR may be used to image the tooth, including imaging through thehole that has been drilled into the tooth, to visualize into the pulpchamber. The scanner may be oriented, automatically or manually, toimage down into the chamber, which may allow visualization of the rootswithin the chamber. The initial drilling into the teeth may be limitedto penetrate the enamel and expose the inner chamber, and visualizinginto the chamber so that regions having different optical properties (atany wavelength, including in particular the near-IR wavelengths) maypenetrate into the chamber despite calcifications and/or infection, toallow imaging of the roots from within the tooth itself. The nervechambers of the root may be identified as being more or less dense thanthe surrounding regions within the dentin and enamel. By removing theroof of the chamber to expose the inner pulp region of the tooth, theintraoral scanner may visualize through the drilled opening to provideadditional volumetric information, including the locations, curvature,and trajectory of the tooth root. Detection of hidden canals andaccessory canals may be facilitated by this additional visualizationinformation. This information may then be used to guide the therapy.

For example, in some variations, the method may include taking, using anintraoral scanner as described herein, a 3D volumetric model of thepatient's teeth either before or after drilling to form an opening intothe target tooth (e.g., for which a root canal will be performed). Thedrilling may be done with or without guidance from an intraoral scanner,as described above. The inner chamber of the tooth may be visualizedusing the intraoral scanner, e.g., through an opening drilled from thecrown of the tooth. The apparatus may then determine the locations ofthe horns of the pulp chamber for the tooth. Any of the methodsdescribed herein may be used in combination with x-ray information.Treatment planning may be performed by the apparatus to determine theshape and/or location of the pulp horns, pulp chamber, and roots to mapout a treatment plan for drilling/tissue removal that avoidsoverthinking or breaching the lateral sides of the tooth. This treatmentplan may then be used to guide the user in drilling on the teeth, and/orfor automating the drilling. In some variations the drill may bedirectly guided by imaging, e.g., using the hybrid drill/intraoralscanner described above. Alternatively or additionally, roboticassistance may be provided using the treatment plan. In some variations,the procedure may be performed manually, and the drilling may be done insmall increments, with visualization between drilling steps to confirmthe treatment path, and avoid over-drilling, as well as confirming thatthe entire region has been drilled and infected pulp removed. Additionalvisualization (including using a contrasting agent) may be used.

In general, any of the methods described herein, including the rootcanal methods described above, may be used with one or more contrastingagents during imaging. For example, contrasting agent may includematerial applied to the outside of the tooth (or into a hole or openingin the tooth, including holes drilled into the tooth). Contrast agentsthat absorb or reflect in the near-IR, or other wavelengths use by theintraoral scanner may be used. Preferably contrast agents may be usedthat are distinguishable at some of the imaged wavelengths, but not allof them, to provide differential imaging. For example, contrast agentsmay be visualizable under white light, but not near-IR; alternatively, acontrast agent may be visualizable under near-IR but not white light, orunder some wavelengths of near-IR but not others for which images aretaken. Contrast agents that preferably attach mix or coat with one ormore targets within the teeth or oral cavity may be used. For example, acontrast agent that selectively binds to one or more of: bacteria,plaque, calculus, gingiva, pulp, etc., may be used. In use, the contrastagent may be applied to the teeth/oral cavity, rinsed, then visualized(or visualized without rinsing). For example, a contrast agent thatabsorbs IR light may be used for inclusion as part of, or mixed in witha material forming, e.g., a dental implant (e.g., to fill a cavity, capa tooth, fill a root canal, etc.,) to create an IR contrasting fillermaterial that may be easily visualized when scanning as describedherein.

Also described herein are methods of determining improvements in softtissue around the teeth using the apparatuses and methods for generating3D volumetric models of the teeth, as described herein. For example agum recession may be monitored and/or quantified, and may be observedover time using these methods and apparatuses. In addition to the directvisualization of plaque and/or calculus as described above, the methodsand apparatuses described herein may also or alternatively detect theeffect on the teeth, including recession of the bone due to plaque andcalculus. Diseased regions, may be visualized directly. In somevariations a contrast agent may be used to provide additional contrastfor the intraoral scanner to detect diseased regions. Scanning of thesurface of the gingiva may identify inflamed and/or discolored regionsthat may be indicative of gum disease. This information may be combinedwith the 3D volumetric modeling of the teeth, including the location ofplaque and/or calculus, as discussed above.

FIGS. 8A and 8B illustrate an example of a monitoring, over time, gum(gingival) recession. In this example, the display may show the 3D modelof the teeth and a comparison between the original scan, and asubsequent scan, taken 2-3 years later. In FIG. 8A, the two scans havebeen aligned and compared, and differences shown by a color indicator,e.g., a heat map. In FIG. 8, darker colors (which may be shown in color,e.g., red) show recession of the gingiva to a greater degree. Thecircled region B in FIG. 8A is shown in greater detail in FIG. 8B forthe later scan. Although FIG. 8 illustrates primarily surface features(e.g., gingiva position), volumetric information may be used to generatethis information, e.g., showing changes in gingiva thickness and/orvascularization, enamel thickness, etc.

In addition to guiding the user and/or dental technician based on thescans (e.g., showing plaque, calculus and/or inflammation inparticular), these methods and apparatuses may be used by the dentalprofessional to rate, rank or quantify the removal of plaque and/orcalculus, either immediately following a treatment, or over time. Thismay provide a metric against which treatments may be judged. The scaninformation may also be used to provide information to the patients,including a map or guide for home treatment, including which areas tofocus on brushing, flossing, etc. The guide may include one or moreimages from the 3D volumetric model, for example. Guidance informationabout what teeth or oral cavity regions to focus home dental care (e.g.,brushing) on may be provided to an electronic toothbrush that may alsohelp guide the patient in brushing based on identified regions.

The methods and apparatuses described herein may also be used withpatient's already having a dental appliance installed on the teeth,including braces, bridges, and the like. For example, in some variationsthe patient may include 3D representations of the dental appliance, andmay provide information to help design or modify future dental devices(e.g., retainers, aligners, braces, etc.).

In particular, the methods and apparatuses described herein may be useto provide very accurate volumetric and surface information about thepatient's teeth that may be useful for treatment planning of any type ofdental treatment. In some variations the methods and apparatusesdescribed herein may be useful for treatment planning of an appliance,such as an aligner or retainer, that is optimally worn in closeproximity to the patient's teeth. For example, a method and/or apparatusthat includes a 3D volumetric scan of the patient's teeth may be used tosubtract out or remove from the 3D model of the teeth, any plaque,calculus and/or food debris that might be present at the time of the 3Dscan. By digitally subtracting out any plaque, calculus, and/or fooddebris present, the volumetric information may be used with a virtualrepresentation of an aligner, retainer, night guard, or other device,and the fit improved prior to fabricating, applying or wearing theapparatus.

The gingival tissue surrounding the teeth, being of different density(or different optical absorption/reflection properties) than the enamel,may also be identified and characterized with greater accuracy, so thatthe junction between the inner contour and the tooth surface can beidentified. By doing this, the shape of the tooth surface beneath thegingival tissue can be accurately characterized so that predictivemodels of tooth movement can have more accurate representations of theteeth as portions of the teeth not initially visible become graduallyexposed as the teeth align. In other words, some parts of the teeth maybe initially obscured by the gingival tissue, but as the teethstraighten, the gingival tissue migrates, and the previously-coveredregions are exposed. By detecting the tooth regions beneath the gingivaltissue accurately, the future state of the teeth after the gingiva hasmigrated can be more accurately modeled.

The methods and apparatuses described herein may also be sued to detect,diagnose, and/or treat disorders of the oral cavity.

For example, the 3D volumetric scanning and modeling methods andapparatuses described herein may be used to detect and/or treat salivarystones (e.g., plugging of the salivary ducts). These glands, which maybe located near the molars and under the patient's tongue, may bescanned using the intraoral scanners described herein. These scans maypenetrate the soft tissue and may detect the hard, stone-like formations(i.e., sialoliths, salivary-gland stones, or duct stones) that arecalcified structures that may form inside a salivary gland or duct andblock the flow of saliva into the mouth. The methods and apparatusesdescribed herein may be used to identify these structures and/or mayguide and/or confirm removal of these stones.

In addition to or instead of the use of the apparatuses and methodsdescried herein to identify, diagnose and/or track regions, includingpre-cavitation caries, crack, etc., the methods and apparatusesdescribed herein may also or alternatively be used to identify andmanipulate regions that have already been modified. For example,fillings, attachments (for attaching an angler, braces, etc.), braces,retainers, etc. and any other structures may be identified within thevolumetric model and/or displayed. For example regions of enamel and/orenamel-like restorations may be displayed differently in the volumetricmodel. These regions will typically have different optical properties,including different scattering/absorption of the near-IR (and in somecases visible light) compared to each other and/or other regions of theoral cavity, including the dentin. Such regions may be manually,automatically or semi-automatically identified, and may be segmentedand/or separately manipulated. For example, in some variations theseregions (e.g., attachments/cement, etc. for an aligner or otherappliance) on the tooth may be identified for removal by the dentalpractitioner, and the 3D volumetric model or data (images) taken from itmay be provided to guide such treatment. They may also or alternativelybe digitally subtracted to provide a better fit for a new appliance onceremoved. A subtracted view may also or alternatively be provided to apatient.

In some variation the internal structural integrity of an artificialdental structure or modification (e.g., dental bond, filling, etc.) maybe determined using the volumetric model(s) described herein. Forexample, a volumetric model may include internal detail of an artificialdental structure, such as the structural detail within a filling, bond,etc., or the interface between the natural tooth (enamel, dentin, etc.),and this information may be presented or shown to the user in detail toallow an assessment (or to allow automatic assessment) of the conditionof such artificial dental structures. This may facilitate their removal,repair and/or replacement.

The 3D volumetric models of the teeth (and method and apparatuses forgenerating them) may also be used as a diagnostic or detection tool forfuture tooth sensitivity. For example, an abfraction is a form ofnon-carious tooth tissue loss that typically occurs along the gingivalmargin. The abfraction lesion may be a mechanical loss of toothstructure that is not caused by tooth decay that may occur in both thedentin and enamel of the tooth. These are believed to be caused byrepetitive stress cycles from the patient's occlusion, and exacerbatedby aggressive brushing. The 3D volumetric models of the teeth enhancedby density analysis of the enamel and dentin near the gingival line mayprovide an early indicator of these lesions. For example, an apparatusmay examine the volumetric model to identify the initial stages offormation for these crescent-shaped lesions. Multiple 3D volumetricmodels taken over time may indicate the rate of progression of theselesions. A system may be configured to automatically or manuallyidentify them; as described above, they may be automatically orsemi-automatically flagged.

Thus, the apparatus and methods may identify and alter the user thatsuch a “hotspot” leading the future tooth sensitivity may be occurring,and may provide for treatment plans to slow, stop or reverse theprogression of the lesion. Tooth sensitivity can result from these smallfractures and the exposed dentin. Detection may be triggered byidentifying the characteristic crescent shape that develops in the moremature lesions, however earlier detection may be made by identifyingregions of thinning in the enamel and/or dentin (e.g., near the gingivalline), which may progress over time. The apparatuses and methods mayflag and/or assign risk based on the actual thickness and/or theprogression of changes in the thickness.

The methods and apparatuses described herein may also be used to detectthe development of acid reflux, based in part on characteristic wearpatterns, and/or changes (e.g., over time) in the enamel thickness ofthe patient. For example, acid reflex while a patient sleeps may resultin the gradual erosion of the patient's teeth in a characteristicpattern (e.g., from the back of the teeth, on the lingual side. Assimilar pattern may develop with bulimia. The volumetric models of thepatient's teeth taken, e.g., by near-IR, may provide an accurate mappingof the enamel density and thicknesses of all of the patient's teeth.Thus, a method of detecting acid reflux (or bulimia) may includedetecting (including detecting over time) characteristic thinning of theenamel of the patient's teeth in the rear, lingual region. The moreproximal, lingual region of the teeth may have an unusually thinner (orthinning) enamel thickness, compared to more anterior (forward) regionson the opposite, buccal, side of the patient's teeth.

The methods and apparatuses described herein may also be used to detectthin enamel regions from occlusal wear due to chronic grinding of thepatient's teeth and/or predict tooth sensitivity that may result fromthis grinding. 3D volumetric models of the patient's teeth may show asnapshot of the occlusal thickness of the patient's enamel and theproximity of the dentin to the occlusal surface. Further, multiple scanstaken over time may show the loss of enamel in the occlusal surface.This was mentioned above as one indicator that may be automatically,manually or semi-automatically marked or flagged. For example, a flagcan be set whenever a region >0.5 mm² develops within 0.5 mm of dentin,and the regions of the digital model highlighted. This allows anyregions which satisfy the flag criteria to be visualized and/ormonitored. Given a patient's age and in some variations gender, as wellas the changes in the enamel thickness over time, an estimate of thewear rate over time may be provided, along with proximity to dentinregions, and thus an estimate or prediction of the tooth sensitivity orpain may be made. Grinding of teeth may also be an indicator of otherissued, including sleep apnea. For example, sleep apnea may also bedetected from 3D volumetric models of the patient's teeth, particularlyover time. Many patients with sleep apnea grind their teeth (e.g., in aforward and/or side to side motion), which may result in a pattern oferosion of the teeth. Thus, the methods and apparatuses described hereinmay be used to help diagnose or confirm sleep apnea.

In general, any of the methods and apparatuses described herein may beused with non-human patients. For example, any of the methods andapparatuses described herein may be used as with veterinary patient's(e.g., animals) to determine, for example, the state of the animalsteeth, including wear on the teeth.

The methods and apparatuses described herein may also be used to providean estimate of risk for the patient in developing fractures in theteeth, and/or the development of tooth sensitivity. For example, a the3D volumetric models of the teeth described herein may be used toidentify malocclusions in the teeth and resulting wear and/or crackingof the teeth, based on the mechanical estimates of the tooth thicknessand wear pattern. Functional information such as chewing pattern andarticulation forces may also be integrated into the assessment. Wearpatterns may be identified and shown as ‘hotspots’ for example on imagesgenerated from the 3D representation of the patient's teeth. This may bedisplayed to the patient as information, including as informationwarning of potential risks. High risk regions may be identified to thepatient along with an explanation of the potential risk.

In general, the methods and apparatuses, and particularly the monitoringand comparison, over time, of 3D volumetric models including informationabout the internal structures of the teeth (e.g., enamel and dentindistribution within the teeth) may be used to identify, monitor,diagnose, and guide treatment of a variety of disorders in addition tothose mentioned above. For example, dentin dysplasia, enamel dysplasia,etc. These methods also allow the identification of multiple differenttypes of enamel within the patient's teeth, including regions havingdifferent amounts hydroxyapatite, amelogenins and/or enamelins, ordifferently organized regions of these, including regions that arehomogenous or non-homogeneous, and that may have different opticalproperties for the near-IR wavelengths used for imaging.

Interactive Display of 3D Model of a Patient's Dental Arch

As already described (and shown in the figures above), the methods andapparatuses described herein may allow a user to virtually scan apatient's dental arch. In particular a 3D model of the patent's dentalarch(s), which may be volumetric, surface, or both (or in somevariations an abstracted or generic model), may be used in conjunctionwith images taken, e.g., using an intraoral scanner, from variouspositions around the dental arch. These images may be images that wereused to generate the 3D model of the dental arch. The images may betagged and/or arranged in the data structure to indicate theircorresponding position or region or angle relative to the 3D dental archmodel. In some variations, the 3D model and the images taken may bemaintained as a data structure, however it is not necessary that the 3Dmodel be included with the images as a single data structure.

For example, FIG. 13 is an example of a data structure that includes oneor more dental arch models 1305 as well as a plurality (e.g., greaterthan 50, greater than 100, greater than 200, greater than 500, greaterthan 750, greater than 1000, greater than 10,000, etc.) of one or more(e.g., sets) of images taken from positions around the patient's dentalarch. In some variations both visible light and near-IR (or near-IR andother modalities) images 1301 may be shown and may share positionalinformation. The positional information typically includes the region ofthe dental arch (e.g., in x, y, z coordinates, such as the coordinate ofa center point of the image relative to the dental arch) from which theimage was taken, as well as the angle (e.g., roll, pitch and/or yaw, orradial coordinates, etc.) relative to the plane of the dental arch(“positional info” 1301). In some variations the scans may be compositesof multiple scans (e.g., averages, blends, etc.) that are combined andstored in the data structure. The 3D model may be formed by virtually“stitching” the scans together to form the 3D model.

The data structure may be stored in a compressed configuration; althoughit may contain a large amount of data, the compression and organizationof the data structure may allow it to be manipulated for display. Forexample, FIG. 12 illustrates one method of interactively displaying a 3Dmodel of a patient's dental arch using a data structure such as the oneschematically shown in FIG. 13.

In FIG. 12, the method includes displaying the 3D model of the patient'sdental arch 1201 and displaying on the 3D model a viewing window 1203.The user may then be allowed to continuously move the two (e.g., eitheror both the viewing window and the 3D model) so that the teeth of thedental arch may be virtually viewed “though” the viewing window ingreater detail in a nearby view 1205. The angle of the viewing window aswell as the location of the viewing window along the dental arch may bechanged by the user, e.g., moving continuously over and/or around the 3Dmodel of the dental arch 1207. As the viewing window/dental arch aremoved relative to each other, a corresponding image (or images), such asa near-IR image, taken at a position relative to the dental archcorresponding to the position of the viewing window, may be identifiedfrom the data structure/data set (e.g., FIG. 13) 1209. The correspondingimage(s) may then be displayed 1211, and this process may be iterativelyrepeated as the viewing window is moved over and along the 3D dentalarch model.

In some variations, the data structure may be configured or arrangedtopographically or in an indexed topographic manner; thus images ofadjacent regions may be linked or ordered in the data structure,simplifying the method.

FIGS. 14A-16C illustrate examples of one variation of a user interfacethat may allow the user to virtually scan a 3D model of the dental arch,showing corresponding images (e.g., near-IR images) as described in FIG.12. As mentioned above, the near-IR images may be viewed by the user toidentify manually (or in some variations automatically) identify one ormore structures/defects and/or actionable dental features, includingdental caries, cracks, wear, etc. The display of corresponding 3D dentalarch model and visible light images of the same regions may both giveperspective and allow for immediate comparison with the patient's teeth,simplifying and powerfully augmenting dental analysis.

For example, in FIG. 14A, the display is shown as a user interface 1400including a dental arch model 1403 (3D dental arch model) reconstructedform scans of the patient's teeth and stored, along with many or all ofthese scans, in a data structure. As already mentioned above, it is notnecessary, but may be helpful, for the 3D dental arch model to beincluded in the data structure with the plurality of images. Further,the 3D dental arch model in this example is constructed from the scansof the patient's teeth, however, is should be clear that the 3D dentalarch model may be non-representative, and yet may be used to select the2D views to be displayed, as described herein. A viewing window 1401,shown as a loop or circle, may be moved over or along the 3D model ofthe dental arch; as the viewing window is moved, each of two imagedisplays 1405, 1407 are updated with images corresponding to theposition (both the region of the dental arch and the angle of the dentalarch relative to the plane of the viewing window. In FIG. 14A the first(upper) image 1405 is a near-IR image and a corresponding (taken at thesame approximate time/location) visible light (e.g., color) image isshown in the bottom image 1407. Alternatively displays are shown inFIGS. 14B and 14C, showing just a single image each; in FIG. 14B anenlarged near-IR display image is shown, while in FIG. 14C a single,enlarged visible light display image of the region corresponding to theimaging window view is shown.

The user interface may also include tools 1409 for manipulating thedisplay (e.g., rotating, moving the dental arch and/or viewing window,modifying, marking, etc., the images and/or 3D model, saving,opening/recalling images, etc.

FIGS. 15A-15B illustrate an example of moving the viewing window overthe teeth and changing/updating the corresponding images. FIG. 15A showsthe image of the dental arch with corresponding near-IR and light imagesas “seen” through the viewing window at a middle region of the dentalarch. In FIG. 15B the dental arch has been rotated by the user (oralternatively, the viewing window has been rotated relative to thedental arch lingually) so that the viewing window is slightly linguallypositioned relative to FIG. 15A; the corresponding views (near IR andvisible light) have been updated in real time to show this change of therelative position of the viewing window.

Similarly, FIGS. 16A-16C shows an example of a 3D model of a patient'slower arch similar to the view shown in FIG. 14A-14C. In use, as theuser scans over and along the dental arch by moving the viewing window(and/or the dental arch relative to the viewing window), the displayimages may change virtually continuously, so that they may update inreal or near-real time. The user may identify features in the near-IRimage(s), including densities changes in the region of normallyIR-transparent enamel, which may indicate carries, cracks, or wearing inthe enamel.

The intraoral scanning system shown in FIGS. 1A-1B may be configured asan intraoral scanning system. Returning to FIG. 1A, the intraoralscanning system 101 includes a hand-held wand 103 having at least oneimage sensor and a light source configured to emit light at a spectralrange within near-infrared (near-IR) range of wavelengths, and a displayoutput (screen 102). The screen may be a touchscreen acting as a userinput device, or the system may include a separate user input device(e.g., keyboard, touchpad, joystick, mouse, track ball, etc.). Asindicated in FIG. 1B, the system may also include one or more processorsthat are operably connected to the hand-held wand, display and userinput device. The one or more processors may include circuitry and/orsoftware and/or firmware configured to: display a three-dimensional (3D)model of a patient's dental arch on the display output; display aviewing window over a portion of the 3D model of the patient's dentalarch on the display output; change a relative position between theviewing window and the 3D model of the patient's dental arch based oninput from the user input device; identify, from both the 3D model ofthe patient's dental arch and a plurality of images of the patient'sdental arch taken from different angles and positions relative to thepatient's dental arch, a near-infrared (near-IR) image taken at an angleand position that approximates a relative angle and position between theviewing window relative and the 3D model of the patient's dental arch;and display the identified near-IR image taken at the angle and positionthat approximates the angle and position between the viewing windowrelative to the 3D model of the patient's dental arch (as shown in FIGS.14A-16C).

Automatic Characterization of Dental Features

Also described herein are methods and apparatuses (e.g., systems,including software) that is configured to use the 3D models, includingbut not limited to the volumetric 3D models, of all or a portion of apatient's dental arch to automatically or semi-automatically identify,confirm and/or characterize one or more dental feature. In particular,these methods and apparatuses may be configured to identify, confirm,and/or characterize one or more actionable dental features that maybenefit from detection and/or treatment. Actionable dental features mayinclude, but are not limited to cracks, gum recess, tartar, hard tissueand soft tissue oral conditions, etc. Enamel thickness may be anotheractionable dental feature. For example, the methods and apparatusesdescribed herein may automatically map enamel thickness (e.g., applycolor map where enamel is lower than x microns thick, where x may bepreset and/or user adjustable). Areas of thin enamel are potential areaswhere caries may exist. Other potential actionable dental features mayinclude discoloration (e.g., discontinuities in color), pits, fissures,evidence of grinding (thinning, including thinning over time),interproximal voids, etc., or any other similar feature that may beindicate or suggestive of where caries are likely to form.

Any of the methods and apparatuses described herein may use multipledifferent images or sets of images of the patient's teeth taken withdifferent imaging modalities are used to detect, analyze and/orcharacterize dental features, and particularly actionable dentalfeatures. The multiple different images or sets of images of thepatient's teeth taken with different imaging modalities may each bereferred to as a “record”. Each record may be a different imagingmodality, such as dental cone beam computed tomography (CBCT) scanning,three dimensional (3D) intra-oral scanning, color scanning (one or moreof: 3D color scanning, surface color scanning, etc.), two-dimensional(2D) color scanning, near-IR scanning (including, but not limited to oneor more of: volumetric near-IR imaging, trans illumination and/orreflective scanning), X-ray (including, but not limited to:cephalometric analysis x-ray scanning, panoramic x-ray scanning, etc.),etc., and may include text or graphic chart information of the patient.

For example, each record may initially be processed independently. Oneor more dental features, and in particular, one or more actionabledental features, may be identified by this initial scan. A single record(e.g., a single imaging modality) may be used first to identify the oneor more actionable dental features, or all of the records, or a subsetof the records may be initially processed to identify the one or moreactionable dental features. The initial identification of the one ormore actionable dental features may be performed manually orautomatically or semi-manually. For example, one or more actionabledental features may be identified automatically; a system as describedherein may review the record (including the one or more images of thepatient's teeth) to flag or identify regions having a characteristicassociated with an actionable dental feature. A system may be trained,using machine learning techniques such as supervised learning techniques(e.g., classification, regression, similarity, etc.), unsupervisedlearning techniques (e.g., density estimation, cluster analysis, etc.),reinforcement learning (e.g., Markov Decision Process techniques, etc.),representation learning techniques and/or principle component analysis,etc., to identify/flag a region of a particular scan in a specifiedmodality that is associated (even loosely associated with) an actionabledental characteristic. Alternatively or additionally, a user (dentalprofessional, technician, etc.) may manually review one or more records(each in a particular imaging modality) and may flag or identify regionssuspected to show an actionable dental characteristic. In asemi-automated configuration the system may initially flag one or moreregions from a record that the user may then review and confirm/reject.

As the one or more regions are identified, they may be flagged and/orstored in a collection of potential actionable dental features. Thelocation may be relative to (e.g., the location on) the originatingrecord, or relative to a reference model (such as the 3D volumetricmodel, as will be described in greater detail below). In some variationsthe collection (e.g., array, data structure, file, etc.) may alsoinclude one or more of the type of potential actionable dental features,the extent of the potential actionable dental features, a grade and/ordegree of the potential actionable dental features, the originatingrecord and/or the imaging modality of the originating record, etc. Insome variations the data structure may be integrated into theoriginating record (or a copy thereof) and may modify the image(s) ofthe originating record, e.g., by include a flag or marker at thelocation of the identified potential actionable dental features and/orany meta text such as the grade and/or degree, etc. The grade and/ordegree may refer to the confidence level or score for the potentialactionable dental feature, including the confidence level or score thatthe identified potential actionable dental features is likely ‘real’.

This initial identification process to identify potential actionabledental features may be performed across multiple records, or it may belimited to a subset of the records (e.g., including just to one of therecords), as mentioned above. In some variations the process may beiteratively performed.

Once one or more potential actionable dental features is identified, itmay be cross-referenced to the other one or more records that use(s)other imaging modalities. Thus, the locations of the one or morepotential actionable dental features may be examined in particulardetail to determine if the same potential actionable dental feature isapparent on these one or more other record. In some variations theentire additional record(s) may be examined during this confirmationportion of the procedure, and any additional potential actionable dentalfeatures from the additional one or more records may be likewise flaggedas a potential actionable dental feature and the same region of thedental arch may be examined for these other potential actionable dentalfeatures (including returning back to records that have already beenreviewed, such as the first or originating record).

Comparison across other records may be guided by translating thelocations of the dental features (including but not limited to thepotential actionable dental features) between the different records. Inparticular, it may be helpful to coordinate the individual dentalrecord(s) begin examined to a model of the patient's dental arch, suchas any of the 3D models, and in particular, the 3D volumetric models,described above. The 3D model of the dental arch may therefore act as akey to translate the locations of the one or more potential actionabledental features and may allow rapid and efficient comparisons betweenthe different records, e.g., different imaging modalities.

Thus, a correlation between each of the different records and, inparticular, a correlation between all or some of the different recordsand a 3D model (e.g., a 3D volumetric model) of the dental arch may beestablished either before or after the initial scan for potentialactionable dental features. Any method of correlating a records andother records and/or a 3D model of the patient's dental arch (or aportion of the dental arch) may be used. For example, one or more easilyrecognizable features (e.g., tooth edge, shape, segmentation, etc.) maybe used to determine landmarks that may translate between the one ormore records and/or the one or more records and the 3D model of thepatient's dental arch. In some variations a translational dataset may becreated that includes a transformation between the records and/orbetween each record and a 3D volumetric model of the patient's dentalarch. For example, a 3D volumetric model of all or a portion of thedental arch may include transformation information for each of the oneor more records allowing transformation of the image(s) of the one ormore records, such as an estimate of the distance and/or orientation ofthe imaging modality relative to the record image(s). This allows bothforward and reverse translation of position between each record and the3D model (e.g. volumetric model).

Thus, a translational dataset may include a 3D model and thetranslational information of each record, so that a portion or region ofa record image (or images) may be projected onto the 3D (translational)model, and the same region then back projected onto a second (or more)record taken with another imaging modality so that the same region maybe examined. In some variations the process may begin with thecollection of all the records and/or an automatic, manual orsemi-automatic registration between all the records. For example, theidentification of individual teeth, palate, gingiva, etc. regions, maybe used to cross-correlate between the different imaging modalitiesand/or the 3D model. In one example, a record including x-ray images maybe correlated with a 3D volumetric model of the patient's teeth bysolving (manually or automatically) for the position and/or orientationof the x-ray camera taking the x-ray images corresponding to the record.The volumetric model may be used to determine and/or confirm thelocation and/or orientation of imaging source for each record. In somevariations the record include explicit (e.g., recorded) informationabout the position and/or orientation and/or imaging parameters used totake the image(s); alternatively or additionally, this information maybe derived. As described above one a pseudo-x-ray image may be generatedand compared to an actual x-ray image of the record.

Once a region corresponding to the region of the potential actionabledental feature from another record is identified, the system or methodmay then determine if the same potential actionable dental features ispresent in this other record. If present, the score (e.g. confidencescore, showing the likelihood that the potential actionable dentalfeatures is real) may be adjusted, e.g., increased if the same or asimilar potential actionable dental feature is present. Depending on thetype of record and the type of potential actionable dental features, theabsence of a potential actionable dental features may result inadjusting the confidence score. For example, the absence of surfacefeatures that are not typically detectable by X-rays, such asdiscoloration, plaque, gum recession, etc., may not result in loweringthe confidence score of the one or more potential actionable dentalfeatures. The more occurrences of finding a potential actionable dentalfeatures a corresponding location between different records (thereforein different modalities), the more likely that the potential actionabledental features really exists.

In comparing the corresponding locations of the one or more potentialactionable dental features the region may be examined manually,automatically or semi-automatically, similar to the originalidentification techniques discussed above. For example, a region of anadditional record corresponding to the location of a potentialactionable dental feature in another record may be examinedautomatically to identify features correlated with the type of potentialactionable dental feature. The system may be trained to recognize thepotential actionable dental feature in the imaging modality of theadditional record and may provide a score indicating the likelihood thatthe potential actionable dental feature is present in this location. Insome variations a user (e.g., technician, dental professional, etc.) maybe presented with an image from the additional record(s) and maymanually indicate the likelihood (yes/no, graded scale, numeric scale,etc.) that the potential actionable dental feature is present in the oneor more additional records.

The final confidence value determined for each potential actionabledental feature may be used by the system: stored, transmitted and/ordisplayed. For example the potential actionable dental feature(s) may bepresented to a dental practitioner in any appropriate manner, includingin a list, on a display, such as on 3D model of the dental arch(including the translational 3D dental model) marked, etc. For example,the system may output a display highlights by color, shape, etc. thelocation of any or all of the potential actionable dental features thatare above a threshold confidence level (so likely to be ‘real’); thedisplay may also include one or more views (from the one or morerecords) of the potential actionable dental feature. The user may set ofadjust the threshold confidence level, including on the fly (e.g.,making the threshold more or less stringent and showing the addition orremoval of potential actionable dental features in response.

FIG. 17 illustrates one example of a method 1700 for characterizingdental features across different imaging modalities as just discussed.In FIG. 17, the method (or a system configured to perform it) mayidentify one or more actionable dental features from one or more records(e.g., one or more images or sets of images of the patient's teeth takenwith different imaging modalities) 1701. For example, the one or moreactionable dental features may be identified by an agent or engine thatis configured to automatically detect one or more actionable dentalfeatures. For example, a system performing the method of FIG. 17 mayinclude an actionable dental feature analysis engine, or may includemultiple actionable dental feature analysis engines each configured toidentify one or more types of actionable dental features or one or moretypes of imaging modality. The engine (e.g., an actionable dentalfeature analysis engine) may be part of a computer system. As usedherein, an engine includes one or more processors or a portion thereof.A portion of one or more processors can include some portion of hardwareless than all of the hardware comprising any given one or moreprocessors, such as a subset of registers, the portion of the processordedicated to one or more threads of a multi-threaded processor, a timeslice during which the processor is wholly or partially dedicated tocarrying out part of the engine's functionality, or the like. As such, afirst engine and a second engine can have one or more dedicatedprocessors or a first engine and a second engine can share one or moreprocessors with one another or other engines. Depending uponimplementation-specific or other considerations, an engine can becentralized or its functionality distributed. An engine can includehardware, firmware, or software embodied in a computer-readable mediumfor execution by the processor. The processor transforms data into newdata using implemented data structures and methods, such as is describedwith reference to the figures herein.

The engines described herein, or the engines through which the systemsand devices described herein can be implemented, can be cloud-basedengines. As used herein, a cloud-based engine is an engine that can runapplications and/or functionalities using a cloud-based computingsystem. All or portions of the applications and/or functionalities canbe distributed across multiple computing devices, and need not berestricted to only one computing device. In some embodiments, thecloud-based engines can execute functionalities and/or modules that endusers access through a web browser or container application withouthaving the functionalities and/or modules installed locally on theend-users' computing devices.

Returning to FIG. 17, the one or more actionable dental features may beidentified from one or more records manually or semi-manually. Forexample, an actionable dental feature analysis engine may initiallyidentify one or more actionable dental features that may then beverified or vetted by a user (e.g., dental technician).

Each actionable dental feature identified may then be flagged and/orrecorded, e.g., in a collection of potential actionable dental features1703. For example a collection of potential actionable dental featuresmay be part of a data structure. Adding the potential actionable dentalfeature(s) to a collection (e.g., data structure) may include recordinga location of the actionable dental feature (e.g., on the originatingrecord) and/or one or more of: type of actionable dental features,grade/degree of confidence of the actionable dental feature, etc. Asused herein, a data structure (which may be included as part of adatastore) is intended to include repositories having any applicableorganization of data, including tables, comma-separated values (CSV)files, traditional databases (e.g., SQL), or other applicable known orconvenient organizational formats. Datastores can be implemented, forexample, as software embodied in a physical computer-readable medium ona specific-purpose machine, in firmware, in hardware, in a combinationthereof, or in an applicable known or convenient device or system.Datastore-associated components, such as database interfaces, can beconsidered “part of” a datastore, part of some other system component,or a combination thereof, though the physical location and othercharacteristics of datastore-associated components is not critical foran understanding of the techniques described herein.

A data structure may be associated with a particular way of storing andorganizing data in a computer so that it can be used efficiently withina given context. Data structures are generally based on the ability of acomputer to fetch and store data at any place in its memory, specifiedby an address, a bit string that can be itself stored in memory andmanipulated by the program. Thus, some data structures are based oncomputing the addresses of data items with arithmetic operations; whileother data structures are based on storing addresses of data itemswithin the structure itself. Many data structures use both principles,sometimes combined in non-trivial ways. The implementation of a datastructure usually entails writing a set of procedures that create andmanipulate instances of that structure. The datastores, describedherein, can be cloud-based datastores. A cloud-based datastore is adatastore that is compatible with cloud-based computing systems andengines.

The identified “putative” actionable dental features (e.g., “potentialactionable dental features”) may be mapped to corresponding physicallocations in one or more other records 1705. As discussed above, in somevariations this may be done using the 3D volumetric model, which maytranslate between the various different types of records (havingdifferent imaging modalities), including projecting a first record ontothe 3D model and then back onto a second region.

Thus, the same corresponding regions in other records may be reviewed todetermine if the potential actionable dental feature is present orsuggested in the additional record(s). In some variations, the methodmay simply collect all of the different corresponding regions forstorage, transmission and/or presentation to a user (e.g., dentalprofessional), e.g., optionally stopping here and allowing the user toreview these flagged region from multiple different imaging modalities(records) in parallel. For example, the potential actionable dentalfeature may be shown for all corresponding views in a side-by-side(e.g., tiled) or sequential view(s).

Alternatively or additionally, the method and/or system mayautomatically or semi-automatically adjust a confidence score for eachof the potential actionable dental features identified. Thus, the systemmay determine if the additional records indicate that the potentialactionable dental feature is more likely to be present or less likely tobe present and may adjust (or determine) the confidence score for eachof the potential actionable dental features, based on the appearance atthe corresponding location in the additional record(s) 1707.

The adjusted confidence levels may then be used to narrow down thepotential actionable dental features. For example, the method or systemmay then filter and/or apply a threshold based on the adjustedconfidence level for each potential actionable dental feature 1709. Insome variation the threshold may be fixed (e.g., confidence level ofgreater than x, where x is a numeric value intermediate between zeroconfidence and 1 (absolute confidence), e.g., 0.1, 0.2, 0.3, 0.4, 0.5,0.6, 0.7, 0.8, etc. In some variations the threshold value may bemanually adjusted by the user and/or may be based on one or morefeatures of the records, such as a quality metric specific to each ofthe records, etc.

Potential actionable dental feature having a confidence level that isabove the threshold value may then be stored, presented and/ortransmitted 1711. For example a user may be presented with a final listand/or display (e.g., using the 3D model) including the flaggedpotential actionable dental features.

Any of the methods and apparatuses (e.g., systems) described herein maybe configured to build a data structure including all or part of themultiple records. For example, a data structure may include a 3Dvolumetric model and all or some of the associated 2D images that wereused to construct it, as described above. In addition, the datastructure may include additional records, such as images taken by X-ray(e.g., panoramic), and/or CBCT, etc. Metadata, e.g., information,including textual information, about the patient and/or images may alsobe included, including optionally patient chart information from thepatient's health/dental records. Alternatively or additionally, anyidentified potential actionable dental features (e.g., findingsidentified from the records) may also be included. The potentialactionable dental features may be used to search/find/mark on the otherrecords.

Typically, when compiling the images (e.g., 2D near-IR images) to buildthe 3D (e.g., volumetric) model, the 2D images that provided informationmay be marked to indicate the significance to the 3D model. For example,2D images may be marked as less relevant or more relevant.

As mentioned above, the collection of potential actionable dentalfeatures, including their confidence level based on their presence inmultiple records may be included as part of the same data structureincluding the 3D model, or it may be separate. The 3D model may bedirectly marked (flagged, coded, etc.) to include the potentialactionable dental features. Thus, the data structure may be acompilation of all of the different records. The combined/compiled datastructure may be referred to as a marked data structure or an actionabledental feature data structure.

Any of the records, including the near-IR 2D images, may be used/scannedto identify the potential actionable dental features. As described abovewhen a suspicious area is identified, either automatically,semi-automatically/semi-manually, or manually (e.g., by a user), in oneof the records, the method or system may then search the correspondingarea of the dental arch on all or some of the other records and concludeif there is a finding. In some variations, the method or apparatus mayupdate the images on all or some of the records (and/or in the combineddata structure) based on the analysis described herein.

In any of the methods and systems described herein, tooth segmentationmay be used on all or some of the records and/or the 3D model to enhanceperformance and usability. Tooth segmentation may be added prior tovolumetric modeling to assist and improve volumetric results and modelquality. For example, the volumetric 3D model may uses the informationof segmentation to potentially enhance performance as additional surface3D information is added. The segmentation information may also assist insegmenting enamel-dentin-lesions to improve auto detection andsuspicious areas marking (e.g., including but not limited to when usingan automatic agent to identify potential actionable dental features).Alternatively or additionally, tooth segmentation may be added to thevolumetric modeling post-processing to assist in segmentingenamel-dentin-lesions to improve auto detection and suspicious areasmarking. For example, segmentation may also or alternatively help withcorrelating the structures between different imaging modalities,including registering findings on volumetric with other modalities toprovide cross-modality visualization. Tooth segmentation may be used toimprove records and cross-modality visualization of clinical findingsand annotations

In any of the methods and apparatuses described herein the confidencelevel indicated may be a quantitative and/or qualitative index. Forexample, a quantitative confidence level “score” may be provided (e.g.,using a number between, for example, 0-100, 0 to 1.0, -100 to 100, orscaled to any range of numeric values). Qualitative indexes may include“high, medium high, medium, medium low, low”, etc. Both qualitative andquantitative confidence levels may be used. A rating system for theconfidence level based on the multiple records as described herein maybe impactful for insurance claims and/or patient communication.

In any of the methods and system described herein, the morphology of thedental arch may be used to help identify the likely areas of interest orpotential issues. Thus, in general, the 3D model (volumetric model) maybe used and/or modified as described herein in order to include theregions of potential actionable dental features. A modified 3D model mayact as a map that visually indicates areas of areas for risk assessment;this may be used, for example, to guide treatment of the patient,including to promote use of sealants, orthodontic treatment or nightguards, etc. In some variations, the modified 3D model may be used toguide a user when additional scans are needed (e.g., when there is a lownumber of scans in the risk areas). As used herein, a modified 3D modelmay include a 3D (e.g., volumetric and/or surface) model that has beenmarked to indicate the locations and/or type and/or confidence level ofpotential actionable dental features. Thus, in general, the use ofadditional data sources to guide users to capture potential areas ofinterest (e.g., when they appear in records, and particularly in recordsother than near-IR/NIRI scan) may help confirm findings of potentialactionable dental features. As mentioned, the results, including amodified 3D model, may help guide the user in scanning or re-scanning(at a future time) the user's dentition. For example, historical scanscan be used as a targeting map while scanning (and to confirm adequatecoverage in those areas). One or more derived images/presentations maybe used in addition or alternatively. For example, tooth segmentationmay be used to generate a tooth chart map (e.g., from the 3D volumetricmodel) that can be used for follow up and auto import into dentalpractice management software (DPMS). For example, individual records maybe lined to match a specified problem to a tooth map.

Any of the methods (including user interfaces) described herein may beimplemented as software, hardware or firmware, and may be described as anon-transitory computer-readable storage medium storing a set ofinstructions capable of being executed by a processor (e.g., computer,tablet, smartphone, etc.), that when executed by the processor causesthe processor to control perform any of the steps, including but notlimited to: displaying, communicating with the user, analyzing,modifying parameters (including timing, frequency, intensity, etc.),determining, alerting, or the like.

When a feature or element is herein referred to as being “on” anotherfeature or element, it can be directly on the other feature or elementor intervening features and/or elements may also be present. Incontrast, when a feature or element is referred to as being “directlyon” another feature or element, there are no intervening features orelements present. It will also be understood that, when a feature orelement is referred to as being “connected”, “attached” or “coupled” toanother feature or element, it can be directly connected, attached orcoupled to the other feature or element or intervening features orelements may be present. In contrast, when a feature or element isreferred to as being “directly connected”, “directly attached” or“directly coupled” to another feature or element, there are nointervening features or elements present. Although described or shownwith respect to one embodiment, the features and elements so describedor shown can apply to other embodiments. It will also be appreciated bythose of skill in the art that references to a structure or feature thatis disposed “adjacent” another feature may have portions that overlap orunderlie the adjacent feature.

Terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of the invention.For example, as used herein, the singular forms “a”, “an” and “the” areintended to include the plural forms as well, unless the context clearlyindicates otherwise. It will be further understood that the terms“comprises” and/or “comprising,” when used in this specification,specify the presence of stated features, steps, operations, elements,and/or components, but do not preclude the presence or addition of oneor more other features, steps, operations, elements, components, and/orgroups thereof. As used herein, the term “and/or” includes any and allcombinations of one or more of the associated listed items and may beabbreviated as “/”.

Spatially relative terms, such as “under”, “below”, “lower”, “over”,“upper” and the like, may be used herein for ease of description todescribe one element or feature's relationship to another element(s) orfeature(s) as illustrated in the figures. It will be understood that thespatially relative terms are intended to encompass differentorientations of the device in use or operation in addition to theorientation depicted in the figures. For example, if a device in thefigures is inverted, elements described as “under” or “beneath” otherelements or features would then be oriented “over” the other elements orfeatures. Thus, the exemplary term “under” can encompass both anorientation of over and under. The device may be otherwise oriented(rotated 90 degrees or at other orientations) and the spatially relativedescriptors used herein interpreted accordingly. Similarly, the terms“upwardly”, “downwardly”, “vertical”, “horizontal” and the like are usedherein for the purpose of explanation only unless specifically indicatedotherwise.

Although the terms “first” and “second” may be used herein to describevarious features/elements (including steps), these features/elementsshould not be limited by these terms, unless the context indicatesotherwise. These terms may be used to distinguish one feature/elementfrom another feature/element. Thus, a first feature/element discussedbelow could be termed a second feature/element, and similarly, a secondfeature/element discussed below could be termed a first feature/elementwithout departing from the teachings of the present invention.

Throughout this specification and the claims which follow, unless thecontext requires otherwise, the word “comprise”, and variations such as“comprises” and “comprising” means various components can be co-jointlyemployed in the methods and articles (e.g., compositions and apparatusesincluding device and methods). For example, the term “comprising” willbe understood to imply the inclusion of any stated elements or steps butnot the exclusion of any other elements or steps.

In general, any of the apparatuses and methods described herein shouldbe understood to be inclusive, but all or a sub-set of the componentsand/or steps may alternatively be exclusive, and may be expressed as“consisting of” or alternatively “consisting essentially of” the variouscomponents, steps, sub-components or sub-steps.

As used herein in the specification and claims, including as used in theexamples and unless otherwise expressly specified, all numbers may beread as if prefaced by the word “about” or “approximately,” even if theterm does not expressly appear. The phrase “about” or “approximately”may be used when describing magnitude and/or position to indicate thatthe value and/or position described is within a reasonable expectedrange of values and/or positions. For example, a numeric value may havea value that is +/−0.1% of the stated value (or range of values), +/−1%of the stated value (or range of values), +/−2% of the stated value (orrange of values), +/−5% of the stated value (or range of values), +/−10%of the stated value (or range of values), etc. Any numerical valuesgiven herein should also be understood to include about or approximatelythat value, unless the context indicates otherwise. For example, if thevalue “10” is disclosed, then “about 10” is also disclosed. Anynumerical range recited herein is intended to include all sub-rangessubsumed therein. It is also understood that when a value is disclosedthat “less than or equal to” the value, “greater than or equal to thevalue” and possible ranges between values are also disclosed, asappropriately understood by the skilled artisan. For example, if thevalue “X” is disclosed the “less than or equal to X” as well as “greaterthan or equal to X” (e.g., where X is a numerical value) is alsodisclosed. It is also understood that the throughout the application,data is provided in a number of different formats, and that this data,represents endpoints and starting points, and ranges for any combinationof the data points. For example, if a particular data point “10” and aparticular data point “15” are disclosed, it is understood that greaterthan, greater than or equal to, less than, less than or equal to, andequal to 10 and 15 are considered disclosed as well as between 10 and15. It is also understood that each unit between two particular unitsare also disclosed. For example, if 10 and 15 are disclosed, then 11,12, 13, and 14 are also disclosed.

Although various illustrative embodiments are described above, any of anumber of changes may be made to various embodiments without departingfrom the scope of the invention as described by the claims. For example,the order in which various described method steps are performed mayoften be changed in alternative embodiments, and in other alternativeembodiments one or more method steps may be skipped altogether. Optionalfeatures of various device and system embodiments may be included insome embodiments and not in others. Therefore, the foregoing descriptionis provided primarily for exemplary purposes and should not beinterpreted to limit the scope of the invention as it is set forth inthe claims.

The examples and illustrations included herein show, by way ofillustration and not of limitation, specific embodiments in which thesubject matter may be practiced. As mentioned, other embodiments may beutilized and derived there from, such that structural and logicalsubstitutions and changes may be made without departing from the scopeof this disclosure. Such embodiments of the inventive subject matter maybe referred to herein individually or collectively by the term“invention” merely for convenience and without intending to voluntarilylimit the scope of this application to any single invention or inventiveconcept, if more than one is, in fact, disclosed. Thus, althoughspecific embodiments have been illustrated and described herein, anyarrangement calculated to achieve the same purpose may be substitutedfor the specific embodiments shown. This disclosure is intended to coverany and all adaptations or variations of various embodiments.Combinations of the above embodiments, and other embodiments notspecifically described herein, will be apparent to those of skill in theart upon reviewing the above description.

What is claimed is:
 1. A method comprising: displaying athree-dimensional (3D) model of a patient's dental arch; displaying aviewing window over at least a portion of the 3D model of the patient'sdental arch; receiving, from a user, a change in a relative positionbetween the viewing window and the 3D model of the patient's dentalarch; identifying, from both the 3D model of the patient's dental archand a plurality of images of a patient's dental arch taken fromdifferent angles and positions relative to the patient's dental arch, anear-infrared (near-IR) image taken at an angle and position thatapproximates a relative angle and position between the viewing windowrelative to the 3D model of the patient's dental arch; and displayingthe identified near-IR image taken at the angle and position thatapproximates the angle and position between the viewing window relativeto the 3D model of the patient's dental arch.
 2. The method of claim 1,wherein the steps of identifying the near-IR image and displaying theidentified near-IR image are done continuously, as the user changes therelative positions between the viewing window and the 3D model of thepatient's dental arch.
 3. The method of claim 1, wherein allowing theuser to change the relative position between the viewing window and the3D model of the patient's dental arch comprises allowing the user tochange one or more of: an angle between a plane of the viewing windowand the patient's dental arch, and a portion of the dental arch adjacentto the viewing window.
 4. The method of claim 1, wherein identifyingcomprises determining a plurality images that approximate the relativeangle and position between the viewing window relative to the 3D modelof the patient's dental arch and averaging the plurality to form theidentified near-IR image.
 5. The method of claim 1, further comprisingcollecting, in a processor, a data set comprising both the 3D model ofthe patient's dental arch and the plurality of images of a patient'sdental arch taken from different angles and positions relative to thepatient's dental arch.
 6. The method of claim 1, wherein displaying theviewing window over a portion of the 3D model of the patient's dentalarch comprises displaying a loop through which the portion of the 3Dmodel of the patient's dental arch may be viewed.
 7. The method of claim1, wherein allowing the user to change the relative position between theviewing window and the 3D model of the patient's dental arch comprisesseparately controlling one or more of: the angle of the 3D model of apatient's dental arch, the rotation of the 3D model of a patient'sdental arch, and the portion of the dental arch adjacent to the viewingwindow.
 8. The method of claim 1, wherein allowing the user to changethe relative position between the viewing window and the 3D model of thepatient's dental arch comprises allowing the user to move the viewingwindow over the 3D model of the dental arch.
 9. The method of claim 1,further comprises identifying, from both the 3D model of the patient'sdental arch and the plurality of images of the patient's dental archtaken from different angles and positions relative to the patient'sdental arch, a second image that approximates the relative angle andposition between the viewing window relative to the 3D model of thepatient's dental arch comprises identifying one of: a visible lightimage and a florescent image; and displaying the second imageconcurrently with the near-IR image.
 10. The method of claim 1, whereindisplaying the identified near-IR image that approximates the angle andposition of the viewing window relative to the displayed 3D modelcomprises displaying the identified near-IR image in a window adjacentor overlapping with the display of the 3D model of the patient's dentalarch.
 11. A method comprising: displaying a three-dimensional (3D) modelof a patient's dental arch; displaying a viewing window over a portionof the 3D model of the patient's dental arch; allowing a user to changea relative position between the viewing window and the 3D model of thepatient's dental arch, including one or more of: an angle between the 3Dmodel of the patient's dental arch relative to a plane of the viewingwindow, and a portion of the dental arch adjacent to the viewing window;and continuously, as the user changes the relative position between theviewing window and the 3D model of the patient's dental arch:identifying, from both the 3D model of the patient's dental arch and aplurality of pairs of images of a patient's dental arch, wherein eachpair of the plurality of pairs includes a first imaging wavelength and asecond imaging wavelength each taken at the same angle and positionrelative to the patient's dental arch, a pair of images taken at anangle and position that approximate the angle and position of theviewing window relative to the displayed 3D model of the patient'sdental arch; and displaying at least one of the identified pair ofimages taken at the angle and position that approximate the angle andposition of the viewing window relative to the displayed 3D model of thepatient's dental arch.
 12. The method of claim 11, wherein displayingcomprises displaying the pair of images in a window adjacent to the 3Dmodel of the patient's dental arch.
 13. The method of claim 11, whereinidentifying comprises determining a plurality of pairs of images thatapproximate the relative angle and position between the viewing windowrelative to the 3D model of the patient's dental arch, and furthercomprises averaging the first imaging wavelength images and averagingthe second imaging wavelength images of the plurality of pairs of imagesto form the identified pairs of images.
 14. The method of claim 11,further comprising receiving, in a processor, the data set comprisingthe 3D model of the patient's dental arch and the plurality of images ofa patient's dental arch taken from different angles and positionsrelative to the patient's dental arch.
 15. The method of claim 11,wherein displaying the viewing window over a portion of the 3D model ofthe patient's dental arch comprises displaying a loop through which theportion of the 3D model of the patient's dental arch may be viewed. 16.The method of claim 11, wherein allowing the user to change the relativeposition between the viewing window and the 3D model of the patient'sdental arch comprises separately controlling a position of the 3D modelof the patient's dental arch relative to the viewing window and aposition of the viewing window relative to the patient's dental arch.17. The method of claim 11, wherein allowing the user to change therelative position between the viewing window and the 3D model of thepatient's dental arch comprises allowing the user to move the viewingwindow over the 3D model of the dental arch.
 18. The method of claim 11,wherein the first imaging wavelength comprises a near-infraredwavelength and the second imaging wavelength comprises one or more of: avisible light wavelength, a florescent wavelength; an x-ray wavelength.19. The method of claim 11, wherein displaying the identified pair ofimages comprises displaying the identified pair of images in windowsadjacent or overlapping with the display of the 3D model of thepatient's dental arch.
 20. A method comprising: displaying athree-dimensional (3D) model of a patient's dental arch; displaying aviewing window over a portion of the 3D model of the patient's dentalarch; allowing a user to change a relative position between the viewingwindow and the 3D model of the patient's dental arch, including one ormore of: an angle between the viewing window and the patient's dentalarch, and a portion of the dental arch adjacent to the viewing window;and continuously, as the user changes the relative positions between theviewing window and the 3D model of the patient's dental arch:identifying, from both the 3D model of the patient's dental arch and aplurality of near-IR images of a patient's dental arch, wherein eachnear-IR image is taken from a different angle and position relative tothe patient's dental arch, a near-IR image taken at an angle andposition that approximates the relative angle and position between theviewing window relative to the 3D model of the patient's dental arch;and displaying the identified near-IR image taken at the angle andposition that approximates the angle and position of the viewing windowrelative to the displayed 3D model of the patient's dental arch.
 21. Anintraoral scanning system, the system comprising: a hand-held wandhaving at least one image sensor and a light source configured to emitlight at a spectral range within near-infrared (near-IR) range ofwavelengths; a display output; a user input device; and one or moreprocessors operably connected to the hand-held wand, display and userinput device, the one or more processors configured to: display athree-dimensional (3D) model of a patient's dental arch on the displayoutput; display a viewing window over a portion of the 3D model of thepatient's dental arch on the display output; change a relative positionbetween the viewing window and the 3D model of the patient's dental archbased on input from the user input device; identify, from both the 3Dmodel of the patient's dental arch and a plurality of images of thepatient's dental arch taken from different angles and positions relativeto the patient's dental arch, a near-infrared (near-IR) image taken atan angle and position that approximates a relative angle and positionbetween the viewing window relative to the 3D model of the patient'sdental arch; and display the identified near-IR image taken at the angleand position that approximates the angle and position between theviewing window relative to the 3D model of the patient's dental arch.22. The intraoral scanning system of claim 21, wherein the one or moreprocessors is configured to receive the plurality of images of thepatient's dental arch taken from different angles and positions relativeto the patient's dental arch.
 23. The intraoral scanning system of claim21, wherein the one or more processors is configured to continuouslyidentify the near-IR image and display the identified near-IR image asthe user changes the relative positions between the viewing window andthe 3D model of the patient's dental arch.
 24. The intraoral scanningsystem of claim 21, wherein the one or more processors is configured tochange the relative position between the viewing window and the 3D modelof the patient's dental arch based on input from the user input deviceby changing one or more of: an angle between a plane of the viewingwindow and the patient's dental arch, and a portion of the dental archadjacent to the viewing window.
 25. The intraoral scanning system ofclaim 21, wherein the one or more processors is configured to identifythe near-IR image taken at an angle and position that approximates arelative angle and position between the viewing window relative to the3D model of the patient's dental arch by determining a plurality imagesthat approximate the relative angle and position between the viewingwindow relative to the 3D model of the patient's dental arch andaveraging the plurality to form the identified near-IR image.
 26. Theintraoral scanning system of claim 21, wherein the one or moreprocessors is configured to display the viewing window over a portion ofthe 3D model of the patient's dental arch comprises displaying as a loopthrough which the portion of the 3D model of the patient's dental archmay be viewed.
 27. The intraoral scanning system of claim 21, whereinthe one or more processors is configured to change the relative positionbetween the viewing window and the 3D model of the patient's dental archbased on input from the user input device by changing one or more of:the angle of the 3D model of a patient's dental arch relative to theviewing window, the rotation of the 3D model of a patient's dental archrelative to the viewing window, and the portion of the dental archadjacent to the viewing window.
 28. The intraoral scanning system ofclaim 21, wherein the one or more processors is configured to change therelative position between the viewing window and the 3D model of thepatient's dental arch based on input from the user input device bychanging the position of the viewing window over the 3D model of thedental arch.
 29. The intraoral scanning system of claim 21, wherein theone or more processors is configured to identify from both the 3D modelof the patient's dental arch and the plurality of images of thepatient's dental arch taken from different angles and positions relativeto the patient's dental arch, a second image that approximates therelative angle and position between the viewing window relative to the3D model of the patient's dental arch that is one or more of: a visiblelight image and a florescent image; and wherein the one or moreprocessors is configured to display the second image concurrently withthe near-IR image.